2024-09-13T11:42:32.016 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-13T11:42:32.035 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T11:42:32.145 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903450 branch: wip-aclamk-cbt-combined description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest} thrashers/force-sync-many workloads/pool-create-delete} email: akupczyk@redhat.com first_in_suite: false job_id: '7903450' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-aclamk-cbt-combined ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos service trim min: 5 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore write v2 random: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(TOO_FEW_PGS\) - \(POOL_APP_NOT_ENABLED\) - slow request - overall HEALTH_ sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-aclamk-cbt-combined sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 owner: scheduled_akupczyk@teuthology priority: 160 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 4159 sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 sleep_before_teardown: 0 subset: 682/13280 suite: rados suite_branch: wip-aclamk-cbt-combined suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 targets: smithi100.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA7IK4oIiP/yLkpdkFlGdYRURPZB2n6+YbXllR3K/X5MgwPuWVHGOmMltw310yG3Z5B5gmroJmL6iSPgbhjKXqo= smithi122.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCpNMI+SeC5pSpCvsbC+7/y+mxh21V3UdMtQYMYajSz/SLni01bqC/nB7sO6MYmgUE7OCFXuz5CxQFAJMD1pXcs= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 store_thrash: true thrash_delay: 1 thrash_many: true - exec: client.0: - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-13_09:43:51 tube: smithi user: akupczyk verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-13T11:42:32.145 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa; will attempt to use it 2024-09-13T11:42:32.150 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks 2024-09-13T11:42:32.150 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-13T11:42:32.152 INFO:teuthology.task.internal:Checking packages... 2024-09-13T11:42:32.177 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd58f73069a8157a9ace12a0247ee3e8497707ea9' 2024-09-13T11:42:32.177 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T11:42:32.177 INFO:teuthology.packaging:ref: None 2024-09-13T11:42:32.177 INFO:teuthology.packaging:tag: None 2024-09-13T11:42:32.177 INFO:teuthology.packaging:branch: wip-aclamk-cbt-combined 2024-09-13T11:42:32.177 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:42:32.177 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-aclamk-cbt-combined 2024-09-13T11:42:32.332 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4697.gd58f7306 2024-09-13T11:42:32.334 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-13T11:42:32.340 INFO:teuthology.task.internal:no buildpackages task found 2024-09-13T11:42:32.341 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-13T11:42:32.387 INFO:teuthology.task.internal:Saving configuration 2024-09-13T11:42:32.401 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-13T11:42:32.421 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-13T11:42:32.453 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi100.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903450', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 11:37:27.403563', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA7IK4oIiP/yLkpdkFlGdYRURPZB2n6+YbXllR3K/X5MgwPuWVHGOmMltw310yG3Z5B5gmroJmL6iSPgbhjKXqo='} 2024-09-13T11:42:32.484 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903450', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 11:37:27.405402', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCpNMI+SeC5pSpCvsbC+7/y+mxh21V3UdMtQYMYajSz/SLni01bqC/nB7sO6MYmgUE7OCFXuz5CxQFAJMD1pXcs='} 2024-09-13T11:42:32.484 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-13T11:42:32.492 INFO:teuthology.task.internal:roles: ubuntu@smithi100.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-13T11:42:32.492 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-13T11:42:32.492 INFO:teuthology.run_tasks:Running task console_log... 2024-09-13T11:42:32.596 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd6fb81fbe0>, signals=[15]) 2024-09-13T11:42:32.596 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-13T11:42:32.604 INFO:teuthology.task.internal:Opening connections... 2024-09-13T11:42:32.604 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi100.front.sepia.ceph.com 2024-09-13T11:42:32.606 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:42:32.692 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T11:42:32.692 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:42:32.780 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-13T11:42:32.787 DEBUG:teuthology.orchestra.run.smithi100:> uname -m 2024-09-13T11:42:32.805 INFO:teuthology.orchestra.run.smithi100.stdout:x86_64 2024-09-13T11:42:32.806 DEBUG:teuthology.orchestra.run.smithi100:> cat /etc/os-release 2024-09-13T11:42:32.863 INFO:teuthology.orchestra.run.smithi100.stdout:NAME="CentOS Stream" 2024-09-13T11:42:32.863 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION="9" 2024-09-13T11:42:32.863 INFO:teuthology.orchestra.run.smithi100.stdout:ID="centos" 2024-09-13T11:42:32.863 INFO:teuthology.orchestra.run.smithi100.stdout:ID_LIKE="rhel fedora" 2024-09-13T11:42:32.863 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION_ID="9" 2024-09-13T11:42:32.863 INFO:teuthology.orchestra.run.smithi100.stdout:PLATFORM_ID="platform:el9" 2024-09-13T11:42:32.863 INFO:teuthology.orchestra.run.smithi100.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T11:42:32.864 INFO:teuthology.orchestra.run.smithi100.stdout:ANSI_COLOR="0;31" 2024-09-13T11:42:32.864 INFO:teuthology.orchestra.run.smithi100.stdout:LOGO="fedora-logo-icon" 2024-09-13T11:42:32.864 INFO:teuthology.orchestra.run.smithi100.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T11:42:32.864 INFO:teuthology.orchestra.run.smithi100.stdout:HOME_URL="https://centos.org/" 2024-09-13T11:42:32.864 INFO:teuthology.orchestra.run.smithi100.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T11:42:32.864 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T11:42:32.864 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T11:42:32.865 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server 2024-09-13T11:42:32.894 DEBUG:teuthology.orchestra.run.smithi122:> uname -m 2024-09-13T11:42:32.911 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2024-09-13T11:42:32.912 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release 2024-09-13T11:42:32.969 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="CentOS Stream" 2024-09-13T11:42:32.969 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="9" 2024-09-13T11:42:32.969 INFO:teuthology.orchestra.run.smithi122.stdout:ID="centos" 2024-09-13T11:42:32.970 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="rhel fedora" 2024-09-13T11:42:32.970 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="9" 2024-09-13T11:42:32.970 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el9" 2024-09-13T11:42:32.970 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T11:42:32.970 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31" 2024-09-13T11:42:32.970 INFO:teuthology.orchestra.run.smithi122.stdout:LOGO="fedora-logo-icon" 2024-09-13T11:42:32.970 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T11:42:32.970 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://centos.org/" 2024-09-13T11:42:32.970 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T11:42:32.970 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T11:42:32.970 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T11:42:32.971 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2024-09-13T11:42:33.002 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-13T11:42:33.011 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-13T11:42:33.035 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-13T11:42:33.035 DEBUG:teuthology.orchestra.run.smithi100:> test '!' -e /home/ubuntu/cephtest 2024-09-13T11:42:33.038 DEBUG:teuthology.orchestra.run.smithi122:> test '!' -e /home/ubuntu/cephtest 2024-09-13T11:42:33.054 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-13T11:42:33.069 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-13T11:42:33.069 DEBUG:teuthology.orchestra.run.smithi100:> test -z $(ls -A /var/lib/ceph) 2024-09-13T11:42:33.096 DEBUG:teuthology.orchestra.run.smithi122:> test -z $(ls -A /var/lib/ceph) 2024-09-13T11:42:33.173 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-13T11:42:33.247 INFO:teuthology.run_tasks:Running task kernel... 2024-09-13T11:42:33.262 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-13T11:42:33.263 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-13T11:42:33.263 DEBUG:teuthology.orchestra.run.smithi100:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T11:42:33.264 DEBUG:teuthology.orchestra.run.smithi122:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T11:42:33.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:42:33.281 DEBUG:teuthology.orchestra.run.smithi100:> uname -r 2024-09-13T11:42:33.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:42:33.282 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-09-13T11:42:33.339 INFO:teuthology.orchestra.run.smithi100.stdout:5.14.0-503.el9.x86_64 2024-09-13T11:42:33.339 INFO:teuthology.task.kernel:Running kernel on smithi100: 5.14.0-503.el9.x86_64 2024-09-13T11:42:33.340 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum install -y kernel 2024-09-13T11:42:33.341 INFO:teuthology.orchestra.run.smithi122.stdout:5.14.0-503.el9.x86_64 2024-09-13T11:42:33.341 INFO:teuthology.task.kernel:Running kernel on smithi122: 5.14.0-503.el9.x86_64 2024-09-13T11:42:33.341 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum install -y kernel 2024-09-13T11:42:35.623 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.2 MB 00:00 2024-09-13T11:42:35.666 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-13T11:42:37.376 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - AppStream 34 MB/s | 20 MB 00:00 2024-09-13T11:42:37.507 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - AppStream 27 MB/s | 20 MB 00:00 2024-09-13T11:42:41.601 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.5 MB 00:00 2024-09-13T11:42:43.092 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.5 MB 00:01 2024-09-13T11:42:43.352 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 19 kB 00:00 2024-09-13T11:42:44.106 INFO:teuthology.orchestra.run.smithi100.stdout:Extra Packages for Enterprise Linux 36 MB/s | 23 MB 00:00 2024-09-13T11:42:44.767 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 19 kB 00:00 2024-09-13T11:42:45.655 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-13T11:42:49.629 INFO:teuthology.orchestra.run.smithi100.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-09-13T11:42:51.036 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-13T11:42:51.602 INFO:teuthology.orchestra.run.smithi100.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T11:42:51.602 INFO:teuthology.orchestra.run.smithi100.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T11:42:51.697 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:42:51.705 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:42:51.705 INFO:teuthology.orchestra.run.smithi100.stdout: Package Architecture Version Repository Size 2024-09-13T11:42:51.705 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:42:51.705 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T11:42:51.705 INFO:teuthology.orchestra.run.smithi100.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T11:42:51.705 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-13T11:42:51.705 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T11:42:51.706 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T11:42:51.706 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T11:42:51.706 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:42:51.706 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T11:42:51.706 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:42:51.706 INFO:teuthology.orchestra.run.smithi100.stdout:Install 4 Packages 2024-09-13T11:42:51.706 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:42:51.706 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 79 M 2024-09-13T11:42:51.706 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 126 M 2024-09-13T11:42:51.706 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T11:42:52.055 INFO:teuthology.orchestra.run.smithi100.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 313 kB/s | 23 kB 00:00 2024-09-13T11:42:53.035 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T11:42:53.035 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T11:42:53.125 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:42:53.137 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:42:53.137 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-09-13T11:42:53.137 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:42:53.137 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T11:42:53.138 INFO:teuthology.orchestra.run.smithi122.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T11:42:53.138 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T11:42:53.138 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T11:42:53.138 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T11:42:53.138 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T11:42:53.138 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:42:53.139 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T11:42:53.139 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:42:53.139 INFO:teuthology.orchestra.run.smithi122.stdout:Install 4 Packages 2024-09-13T11:42:53.139 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:42:53.139 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 79 M 2024-09-13T11:42:53.139 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 126 M 2024-09-13T11:42:53.139 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T11:42:53.479 INFO:teuthology.orchestra.run.smithi122.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 263 kB/s | 23 kB 00:00 2024-09-13T11:42:55.688 INFO:teuthology.orchestra.run.smithi100.stdout:(2/4): kernel-modules-core-5.14.0-505.el9.x86_6 7.9 MB/s | 29 MB 00:03 2024-09-13T11:42:55.930 INFO:teuthology.orchestra.run.smithi100.stdout:(3/4): kernel-core-5.14.0-505.el9.x86_64.rpm 4.0 MB/s | 16 MB 00:03 2024-09-13T11:42:56.630 INFO:teuthology.orchestra.run.smithi100.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 7.4 MB/s | 35 MB 00:04 2024-09-13T11:42:56.630 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:42:56.631 INFO:teuthology.orchestra.run.smithi100.stdout:Total 16 MB/s | 79 MB 00:04 2024-09-13T11:42:56.921 INFO:teuthology.orchestra.run.smithi122.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 4.4 MB/s | 16 MB 00:03 2024-09-13T11:42:57.315 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T11:42:57.684 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T11:42:57.685 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T11:42:58.271 INFO:teuthology.orchestra.run.smithi122.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 6.0 MB/s | 29 MB 00:04 2024-09-13T11:42:58.643 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T11:42:58.644 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T11:42:58.755 INFO:teuthology.orchestra.run.smithi122.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 6.4 MB/s | 35 MB 00:05 2024-09-13T11:42:58.755 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:42:58.755 INFO:teuthology.orchestra.run.smithi122.stdout:Total 14 MB/s | 79 MB 00:05 2024-09-13T11:42:59.437 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T11:42:59.777 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T11:42:59.777 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T11:43:00.754 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T11:43:00.755 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T11:43:01.107 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T11:43:01.748 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T11:43:01.831 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:43:03.198 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T11:43:03.882 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T11:43:03.967 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:43:04.628 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:43:04.769 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:43:06.762 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:43:06.902 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:43:09.576 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:43:09.611 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:43:11.727 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:43:12.002 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:43:14.477 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:43:16.835 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:43:52.844 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:43:52.883 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:43:53.236 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:43:53.237 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T11:43:53.237 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:43:53.237 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:43:54.049 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:43:54.049 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:43:54.049 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T11:43:54.049 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T11:43:54.050 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T11:43:54.050 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T11:43:54.050 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T11:43:54.050 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:43:54.050 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:43:54.389 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T11:43:54.484 INFO:teuthology.orchestra.run.smithi100.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T11:43:54.485 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T11:43:54.485 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T11:43:54.485 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T11:43:54.485 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T11:43:54.485 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-13T11:43:54.485 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi100.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T11:43:54.485 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T11:43:54.543 INFO:teuthology.orchestra.run.smithi100.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T11:43:54.544 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T11:43:54.544 DEBUG:teuthology.orchestra.run.smithi100:> sudo rpm -qi grub2-tools 2024-09-13T11:43:54.575 INFO:teuthology.orchestra.run.smithi100.stdout:Name : grub2-tools 2024-09-13T11:43:54.576 INFO:teuthology.orchestra.run.smithi100.stdout:Epoch : 1 2024-09-13T11:43:54.576 INFO:teuthology.orchestra.run.smithi100.stdout:Version : 2.06 2024-09-13T11:43:54.576 INFO:teuthology.orchestra.run.smithi100.stdout:Release : 82.el9 2024-09-13T11:43:54.576 INFO:teuthology.orchestra.run.smithi100.stdout:Architecture: x86_64 2024-09-13T11:43:54.576 INFO:teuthology.orchestra.run.smithi100.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T11:43:54.576 INFO:teuthology.orchestra.run.smithi100.stdout:Group : Unspecified 2024-09-13T11:43:54.576 INFO:teuthology.orchestra.run.smithi100.stdout:Size : 8274670 2024-09-13T11:43:54.576 INFO:teuthology.orchestra.run.smithi100.stdout:License : GPLv3+ 2024-09-13T11:43:54.576 INFO:teuthology.orchestra.run.smithi100.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T11:43:54.576 INFO:teuthology.orchestra.run.smithi100.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T11:43:54.576 INFO:teuthology.orchestra.run.smithi100.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T11:43:54.576 INFO:teuthology.orchestra.run.smithi100.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T11:43:54.576 INFO:teuthology.orchestra.run.smithi100.stdout:Packager : builder@centos.org 2024-09-13T11:43:54.577 INFO:teuthology.orchestra.run.smithi100.stdout:Vendor : CentOS 2024-09-13T11:43:54.577 INFO:teuthology.orchestra.run.smithi100.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T11:43:54.577 INFO:teuthology.orchestra.run.smithi100.stdout:Summary : Support tools for GRUB. 2024-09-13T11:43:54.577 INFO:teuthology.orchestra.run.smithi100.stdout:Description : 2024-09-13T11:43:54.577 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:43:54.577 INFO:teuthology.orchestra.run.smithi100.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T11:43:54.577 INFO:teuthology.orchestra.run.smithi100.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T11:43:54.577 INFO:teuthology.orchestra.run.smithi100.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T11:43:54.577 INFO:teuthology.orchestra.run.smithi100.stdout:hardware devices. 2024-09-13T11:43:54.577 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:43:54.577 INFO:teuthology.orchestra.run.smithi100.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T11:43:54.578 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T11:43:54.578 INFO:teuthology.task.kernel:Updating grub on smithi100 to boot 5.14.0-505.el9.x86_64 2024-09-13T11:43:54.578 DEBUG:teuthology.orchestra.run.smithi100:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T11:43:55.244 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:43:55.289 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:43:55.314 INFO:teuthology.orchestra.run.smithi100.stderr:Generating grub configuration file ... 2024-09-13T11:43:55.597 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:43:55.597 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T11:43:55.597 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:43:55.597 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:43:56.475 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:43:56.475 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:43:56.475 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T11:43:56.475 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T11:43:56.475 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T11:43:56.475 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T11:43:56.475 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T11:43:56.475 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:43:56.475 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:43:56.656 INFO:teuthology.orchestra.run.smithi100.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T11:43:56.688 INFO:teuthology.orchestra.run.smithi100.stderr:done 2024-09-13T11:43:56.690 DEBUG:teuthology.orchestra.run.smithi100:> mktemp 2024-09-13T11:43:56.706 INFO:teuthology.orchestra.run.smithi100.stdout:/tmp/tmp.BqT1O1Jy9g 2024-09-13T11:43:56.706 DEBUG:teuthology.orchestra.run.smithi100:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.BqT1O1Jy9g 2024-09-13T11:43:56.773 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0666 /tmp/tmp.BqT1O1Jy9g 2024-09-13T11:43:56.821 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T11:43:56.897 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T11:43:56.897 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T11:43:56.897 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T11:43:56.897 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T11:43:56.897 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T11:43:56.897 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-13T11:43:56.897 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi122.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T11:43:56.898 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T11:43:56.914 DEBUG:teuthology.orchestra.remote:smithi100:/tmp/tmp.BqT1O1Jy9g is 6KB 2024-09-13T11:43:56.956 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T11:43:56.956 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T11:43:56.956 DEBUG:teuthology.orchestra.run.smithi122:> sudo rpm -qi grub2-tools 2024-09-13T11:43:56.963 DEBUG:teuthology.orchestra.run.smithi100:> rm -fr /tmp/tmp.BqT1O1Jy9g 2024-09-13T11:43:56.977 DEBUG:teuthology.orchestra.run.smithi100:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T11:43:56.988 INFO:teuthology.orchestra.run.smithi122.stdout:Name : grub2-tools 2024-09-13T11:43:56.988 INFO:teuthology.orchestra.run.smithi122.stdout:Epoch : 1 2024-09-13T11:43:56.988 INFO:teuthology.orchestra.run.smithi122.stdout:Version : 2.06 2024-09-13T11:43:56.988 INFO:teuthology.orchestra.run.smithi122.stdout:Release : 82.el9 2024-09-13T11:43:56.988 INFO:teuthology.orchestra.run.smithi122.stdout:Architecture: x86_64 2024-09-13T11:43:56.989 INFO:teuthology.orchestra.run.smithi122.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T11:43:56.989 INFO:teuthology.orchestra.run.smithi122.stdout:Group : Unspecified 2024-09-13T11:43:56.989 INFO:teuthology.orchestra.run.smithi122.stdout:Size : 8274670 2024-09-13T11:43:56.989 INFO:teuthology.orchestra.run.smithi122.stdout:License : GPLv3+ 2024-09-13T11:43:56.989 INFO:teuthology.orchestra.run.smithi122.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T11:43:56.989 INFO:teuthology.orchestra.run.smithi122.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T11:43:56.989 INFO:teuthology.orchestra.run.smithi122.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T11:43:56.989 INFO:teuthology.orchestra.run.smithi122.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T11:43:56.989 INFO:teuthology.orchestra.run.smithi122.stdout:Packager : builder@centos.org 2024-09-13T11:43:56.989 INFO:teuthology.orchestra.run.smithi122.stdout:Vendor : CentOS 2024-09-13T11:43:56.989 INFO:teuthology.orchestra.run.smithi122.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T11:43:56.989 INFO:teuthology.orchestra.run.smithi122.stdout:Summary : Support tools for GRUB. 2024-09-13T11:43:56.989 INFO:teuthology.orchestra.run.smithi122.stdout:Description : 2024-09-13T11:43:56.989 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:43:56.990 INFO:teuthology.orchestra.run.smithi122.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T11:43:56.990 INFO:teuthology.orchestra.run.smithi122.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T11:43:56.990 INFO:teuthology.orchestra.run.smithi122.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T11:43:56.990 INFO:teuthology.orchestra.run.smithi122.stdout:hardware devices. 2024-09-13T11:43:56.990 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:43:56.990 INFO:teuthology.orchestra.run.smithi122.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T11:43:56.991 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T11:43:56.991 INFO:teuthology.task.kernel:Updating grub on smithi122 to boot 5.14.0-505.el9.x86_64 2024-09-13T11:43:56.991 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T11:43:57.047 INFO:teuthology.orchestra.run.smithi100.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T11:43:57.047 INFO:teuthology.orchestra.run.smithi100.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T11:43:57.047 INFO:teuthology.orchestra.run.smithi100.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T11:43:57.047 INFO:teuthology.orchestra.run.smithi100.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T11:43:57.049 DEBUG:teuthology.orchestra.run.smithi100:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T11:43:57.209 DEBUG:teuthology.orchestra.run.smithi100:> sudo shutdown -r now 2024-09-13T11:43:57.653 INFO:teuthology.orchestra.run.smithi122.stderr:Generating grub configuration file ... 2024-09-13T11:43:58.975 INFO:teuthology.orchestra.run.smithi122.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T11:43:58.997 INFO:teuthology.orchestra.run.smithi122.stderr:done 2024-09-13T11:43:59.003 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2024-09-13T11:43:59.019 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.IuNiMN3Qmb 2024-09-13T11:43:59.019 DEBUG:teuthology.orchestra.run.smithi122:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.IuNiMN3Qmb 2024-09-13T11:43:59.087 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.IuNiMN3Qmb 2024-09-13T11:43:59.231 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.IuNiMN3Qmb is 6KB 2024-09-13T11:43:59.252 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.IuNiMN3Qmb 2024-09-13T11:43:59.267 DEBUG:teuthology.orchestra.run.smithi122:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T11:43:59.330 INFO:teuthology.orchestra.run.smithi122.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T11:43:59.331 INFO:teuthology.orchestra.run.smithi122.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T11:43:59.331 INFO:teuthology.orchestra.run.smithi122.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T11:43:59.331 INFO:teuthology.orchestra.run.smithi122.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T11:43:59.332 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T11:43:59.486 DEBUG:teuthology.orchestra.run.smithi122:> sudo shutdown -r now 2024-09-13T11:44:27.237 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-13T11:44:27.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-13T11:44:27.238 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:44:29.491 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-13T11:44:29.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T11:44:29.492 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:44:45.629 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.100 2024-09-13T11:44:47.933 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2024-09-13T11:44:54.637 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-13T11:44:54.638 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:44:56.936 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T11:44:56.937 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:44:57.725 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.100 2024-09-13T11:44:59.997 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2024-09-13T11:45:09.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-13T11:45:09.736 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:45:12.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T11:45:12.001 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:45:28.125 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.100 2024-09-13T11:45:30.429 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2024-09-13T11:45:43.137 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-13T11:45:43.138 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:45:43.482 DEBUG:teuthology.orchestra.run.smithi100:> true 2024-09-13T11:45:43.899 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-13T11:45:43.899 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-13T11:45:43.899 DEBUG:teuthology.orchestra.run.smithi100:> uname -r 2024-09-13T11:45:43.914 INFO:teuthology.orchestra.run.smithi100.stdout:5.14.0-505.el9.x86_64 2024-09-13T11:45:43.914 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T11:45:43.915 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T11:45:43.915 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T11:45:44.916 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-13T11:45:44.916 DEBUG:teuthology.orchestra.run.smithi100:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T11:45:45.027 INFO:teuthology.orchestra.run.smithi100.stdout:ttyS1 2024-09-13T11:45:45.054 DEBUG:teuthology.parallel:result is None 2024-09-13T11:45:45.429 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T11:45:45.430 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:45:45.724 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-09-13T11:45:46.169 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T11:45:46.169 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-13T11:45:46.169 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-09-13T11:45:46.184 INFO:teuthology.orchestra.run.smithi122.stdout:5.14.0-505.el9.x86_64 2024-09-13T11:45:46.184 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T11:45:46.184 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T11:45:46.184 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T11:45:47.185 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-13T11:45:47.185 DEBUG:teuthology.orchestra.run.smithi122:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T11:45:47.290 INFO:teuthology.orchestra.run.smithi122.stdout:ttyS1 2024-09-13T11:45:47.311 DEBUG:teuthology.parallel:result is None 2024-09-13T11:45:47.312 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-13T11:45:47.319 INFO:teuthology.task.internal:Creating test directory... 2024-09-13T11:45:47.320 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T11:45:47.322 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T11:45:47.359 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-13T11:45:47.366 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-13T11:45:47.408 INFO:teuthology.task.internal:Creating archive directory... 2024-09-13T11:45:47.409 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T11:45:47.411 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T11:45:47.530 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-13T11:45:47.537 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-13T11:45:47.537 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T11:45:47.540 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T11:45:47.615 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:45:47.623 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:45:47.625 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:45:47.646 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:45:47.648 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-13T11:45:47.656 INFO:teuthology.task.internal:Configuring sudo... 2024-09-13T11:45:47.656 DEBUG:teuthology.orchestra.run.smithi100:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T11:45:47.658 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T11:45:47.716 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-13T11:45:47.729 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-13T11:45:47.729 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T11:45:47.732 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T11:45:47.781 DEBUG:teuthology.orchestra.run.smithi100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:45:47.843 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:45:47.878 DEBUG:teuthology.orchestra.run.smithi100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:45:47.969 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:45:47.999 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T11:45:47.999 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T11:45:48.065 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:45:48.117 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:45:48.341 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:45:48.391 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:45:48.424 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T11:45:48.424 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T11:45:48.491 DEBUG:teuthology.orchestra.run.smithi100:> sudo service rsyslog restart 2024-09-13T11:45:48.494 DEBUG:teuthology.orchestra.run.smithi122:> sudo service rsyslog restart 2024-09-13T11:45:48.548 INFO:teuthology.orchestra.run.smithi100.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T11:45:48.575 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T11:45:49.019 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-13T11:45:49.028 INFO:teuthology.task.internal:Starting timer... 2024-09-13T11:45:49.028 INFO:teuthology.run_tasks:Running task pcp... 2024-09-13T11:45:49.062 INFO:teuthology.run_tasks:Running task selinux... 2024-09-13T11:45:49.111 DEBUG:teuthology.orchestra.run.smithi100:> sudo service auditd rotate 2024-09-13T11:45:49.176 INFO:teuthology.orchestra.run.smithi100.stdout:Rotating logs: 2024-09-13T11:45:49.179 DEBUG:teuthology.orchestra.run.smithi122:> sudo service auditd rotate 2024-09-13T11:45:49.223 INFO:teuthology.orchestra.run.smithi122.stdout:Rotating logs: 2024-09-13T11:45:49.225 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-13T11:45:49.225 DEBUG:teuthology.orchestra.run.smithi100:> /usr/sbin/getenforce 2024-09-13T11:45:49.271 INFO:teuthology.orchestra.run.smithi100.stdout:Permissive 2024-09-13T11:45:49.271 DEBUG:teuthology.orchestra.run.smithi122:> /usr/sbin/getenforce 2024-09-13T11:45:49.301 INFO:teuthology.orchestra.run.smithi122.stdout:Permissive 2024-09-13T11:45:49.302 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi100.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi122.front.sepia.ceph.com': 'permissive'} 2024-09-13T11:45:49.302 DEBUG:teuthology.orchestra.run.smithi100:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T11:45:49.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:45:49.331 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T11:45:49.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:45:49.368 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-13T11:45:49.368 DEBUG:teuthology.orchestra.run.smithi100:> sudo /usr/sbin/setenforce permissive 2024-09-13T11:45:49.412 DEBUG:teuthology.orchestra.run.smithi122:> sudo /usr/sbin/setenforce permissive 2024-09-13T11:45:49.460 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-13T11:45:49.471 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-13T11:45:49.488 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-13T11:45:49.489 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi100.front.sepia.ceph.com,smithi122.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-13T11:49:34.599 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi100.front.sepia.ceph.com'), Remote(name='ubuntu@smithi122.front.sepia.ceph.com')] 2024-09-13T11:49:34.600 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-13T11:49:34.602 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:49:34.727 DEBUG:teuthology.orchestra.run.smithi100:> true 2024-09-13T11:49:34.808 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-13T11:49:34.808 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T11:49:34.808 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:49:34.890 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-09-13T11:49:34.926 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T11:49:34.926 INFO:teuthology.run_tasks:Running task clock... 2024-09-13T11:49:34.936 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-13T11:49:34.937 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T11:49:34.937 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:49:34.939 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T11:49:34.939 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:49:34.966 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T11:49:34.981 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T11:49:34.997 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T11:49:35.004 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: ntpd: command not found 2024-09-13T11:49:35.011 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T11:49:35.015 INFO:teuthology.orchestra.run.smithi100.stdout:506 Cannot talk to daemon 2024-09-13T11:49:35.030 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T11:49:35.035 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: ntpd: command not found 2024-09-13T11:49:35.044 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T11:49:35.046 INFO:teuthology.orchestra.run.smithi122.stdout:506 Cannot talk to daemon 2024-09-13T11:49:35.060 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T11:49:35.074 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T11:49:35.103 INFO:teuthology.orchestra.run.smithi100.stderr:bash: line 1: ntpq: command not found 2024-09-13T11:49:35.110 INFO:teuthology.orchestra.run.smithi100.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T11:49:35.110 INFO:teuthology.orchestra.run.smithi100.stdout:=============================================================================== 2024-09-13T11:49:35.110 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:49:35.110 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:49:35.110 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:49:35.110 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:49:35.125 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-09-13T11:49:35.129 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T11:49:35.129 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-09-13T11:49:35.129 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:49:35.129 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:49:35.129 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:49:35.129 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:49:35.130 INFO:teuthology.run_tasks:Running task install... 2024-09-13T11:49:35.140 DEBUG:teuthology.task.install:project ceph 2024-09-13T11:49:35.140 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}} 2024-09-13T11:49:35.141 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'} 2024-09-13T11:49:35.141 INFO:teuthology.task.install:Using flavor: default 2024-09-13T11:49:35.153 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-13T11:49:35.153 INFO:teuthology.task.install:extra packages: [] 2024-09-13T11:49:35.154 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T11:49:35.155 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:49:35.156 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T11:49:35.157 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:49:35.310 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/ 2024-09-13T11:49:35.310 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4697.gd58f7306 2024-09-13T11:49:35.326 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/ 2024-09-13T11:49:35.326 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4697.gd58f7306 2024-09-13T11:49:35.436 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T11:49:35.436 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T11:49:35.436 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T11:49:35.447 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T11:49:35.447 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T11:49:35.448 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T11:49:35.464 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T11:49:35.465 DEBUG:teuthology.orchestra.run.smithi100:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d58f73069a8157a9ace12a0247ee3e8497707ea9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T11:49:35.474 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T11:49:35.474 DEBUG:teuthology.orchestra.run.smithi122:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d58f73069a8157a9ace12a0247ee3e8497707ea9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T11:49:35.531 DEBUG:teuthology.orchestra.run.smithi100:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T11:49:35.541 DEBUG:teuthology.orchestra.run.smithi122:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T11:49:35.606 DEBUG:teuthology.orchestra.run.smithi100:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T11:49:35.617 DEBUG:teuthology.orchestra.run.smithi122:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T11:49:35.670 INFO:teuthology.orchestra.run.smithi100.stdout:check_obsoletes = 1 2024-09-13T11:49:35.672 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum clean all 2024-09-13T11:49:35.682 INFO:teuthology.orchestra.run.smithi122.stdout:check_obsoletes = 1 2024-09-13T11:49:35.683 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum clean all 2024-09-13T11:49:35.893 INFO:teuthology.orchestra.run.smithi100.stdout:45 files removed 2024-09-13T11:49:35.908 INFO:teuthology.orchestra.run.smithi122.stdout:45 files removed 2024-09-13T11:49:35.920 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-radosgw 2024-09-13T11:49:35.935 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-radosgw 2024-09-13T11:49:36.476 INFO:teuthology.orchestra.run.smithi122.stdout:ceph packages for x86_64 326 kB/s | 84 kB 00:00 2024-09-13T11:49:36.485 INFO:teuthology.orchestra.run.smithi100.stdout:ceph packages for x86_64 297 kB/s | 84 kB 00:00 2024-09-13T11:49:36.777 INFO:teuthology.orchestra.run.smithi122.stdout:ceph noarch packages 94 kB/s | 24 kB 00:00 2024-09-13T11:49:36.777 INFO:teuthology.orchestra.run.smithi100.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-09-13T11:49:37.044 INFO:teuthology.orchestra.run.smithi100.stdout:ceph source packages 8.3 kB/s | 1.9 kB 00:00 2024-09-13T11:49:37.052 INFO:teuthology.orchestra.run.smithi122.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-13T11:49:37.527 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.2 MB 00:00 2024-09-13T11:49:37.761 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-09-13T11:49:57.886 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 20 MB 00:19 2024-09-13T11:49:59.110 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - AppStream 1.0 MB/s | 20 MB 00:20 2024-09-13T11:50:02.438 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - CRB 8.6 MB/s | 6.5 MB 00:00 2024-09-13T11:50:03.428 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-13T11:50:03.947 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - Extras packages 65 kB/s | 19 kB 00:00 2024-09-13T11:50:04.734 INFO:teuthology.orchestra.run.smithi100.stdout:Extra Packages for Enterprise Linux 34 MB/s | 23 MB 00:00 2024-09-13T11:50:05.078 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 19 kB 00:00 2024-09-13T11:50:06.349 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-13T11:50:10.315 INFO:teuthology.orchestra.run.smithi100.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-13T11:50:11.789 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-09-13T11:50:12.699 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:50:12.702 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:50:12.702 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-13T11:50:12.702 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:50:12.702 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T11:50:12.702 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-radosgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 11 M 2024-09-13T11:50:12.703 INFO:teuthology.orchestra.run.smithi100.stdout:Upgrading: 2024-09-13T11:50:12.703 INFO:teuthology.orchestra.run.smithi100.stdout: librados2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.5 M 2024-09-13T11:50:12.703 INFO:teuthology.orchestra.run.smithi100.stdout: librbd1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T11:50:12.703 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-13T11:50:12.703 INFO:teuthology.orchestra.run.smithi100.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T11:50:12.703 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-base x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T11:50:12.703 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 22 M 2024-09-13T11:50:12.703 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-selinux x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 25 k 2024-09-13T11:50:12.703 INFO:teuthology.orchestra.run.smithi100.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T11:50:12.703 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T11:50:12.703 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T11:50:12.703 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 755 k 2024-09-13T11:50:12.704 INFO:teuthology.orchestra.run.smithi100.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T11:50:12.704 INFO:teuthology.orchestra.run.smithi100.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T11:50:12.704 INFO:teuthology.orchestra.run.smithi100.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T11:50:12.704 INFO:teuthology.orchestra.run.smithi100.stdout: libradosstriper1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 508 k 2024-09-13T11:50:12.704 INFO:teuthology.orchestra.run.smithi100.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T11:50:12.704 INFO:teuthology.orchestra.run.smithi100.stdout: librgw2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T11:50:12.704 INFO:teuthology.orchestra.run.smithi100.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T11:50:12.704 INFO:teuthology.orchestra.run.smithi100.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T11:50:12.704 INFO:teuthology.orchestra.run.smithi100.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T11:50:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-argparse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 45 k 2024-09-13T11:50:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 143 k 2024-09-13T11:50:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cephfs x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T11:50:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T11:50:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rados x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 321 k 2024-09-13T11:50:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 305 k 2024-09-13T11:50:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 99 k 2024-09-13T11:50:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-13T11:50:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-13T11:50:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T11:50:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T11:50:12.705 INFO:teuthology.orchestra.run.smithi100.stdout:Installing weak dependencies: 2024-09-13T11:50:12.706 INFO:teuthology.orchestra.run.smithi100.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-13T11:50:12.706 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:50:12.706 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T11:50:12.706 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:50:12.706 INFO:teuthology.orchestra.run.smithi100.stdout:Install 30 Packages 2024-09-13T11:50:12.706 INFO:teuthology.orchestra.run.smithi100.stdout:Upgrade 2 Packages 2024-09-13T11:50:12.706 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:50:12.707 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 62 M 2024-09-13T11:50:12.707 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T11:50:13.608 INFO:teuthology.orchestra.run.smithi100.stdout:(1/32): ceph-base-19.3.0-4697.gd58f7306.el9.x86 10 MB/s | 5.5 MB 00:00 2024-09-13T11:50:13.658 INFO:teuthology.orchestra.run.smithi100.stdout:(2/32): ceph-selinux-19.3.0-4697.gd58f7306.el9. 504 kB/s | 25 kB 00:00 2024-09-13T11:50:13.833 INFO:teuthology.orchestra.run.smithi100.stdout:(3/32): ceph-radosgw-19.3.0-4697.gd58f7306.el9. 15 MB/s | 11 MB 00:00 2024-09-13T11:50:13.867 INFO:teuthology.orchestra.run.smithi100.stdout:(4/32): libcephfs2-19.3.0-4697.gd58f7306.el9.x8 3.5 MB/s | 755 kB 00:00 2024-09-13T11:50:13.900 INFO:teuthology.orchestra.run.smithi100.stdout:(5/32): libradosstriper1-19.3.0-4697.gd58f7306. 7.4 MB/s | 508 kB 00:00 2024-09-13T11:50:13.959 INFO:teuthology.orchestra.run.smithi100.stdout:(6/32): python3-ceph-argparse-19.3.0-4697.gd58f 773 kB/s | 45 kB 00:00 2024-09-13T11:50:14.051 INFO:teuthology.orchestra.run.smithi100.stdout:(7/32): python3-ceph-common-19.3.0-4697.gd58f73 1.5 MB/s | 143 kB 00:00 2024-09-13T11:50:14.126 INFO:teuthology.orchestra.run.smithi100.stdout:(8/32): librgw2-19.3.0-4697.gd58f7306.el9.x86_6 21 MB/s | 5.5 MB 00:00 2024-09-13T11:50:14.140 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:50:14.148 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:50:14.148 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T11:50:14.148 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:50:14.148 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T11:50:14.148 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 11 M 2024-09-13T11:50:14.149 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrading: 2024-09-13T11:50:14.149 INFO:teuthology.orchestra.run.smithi122.stdout: librados2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.5 M 2024-09-13T11:50:14.149 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T11:50:14.149 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T11:50:14.149 INFO:teuthology.orchestra.run.smithi122.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T11:50:14.149 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T11:50:14.149 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 22 M 2024-09-13T11:50:14.149 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 25 k 2024-09-13T11:50:14.149 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T11:50:14.149 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T11:50:14.149 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T11:50:14.150 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 755 k 2024-09-13T11:50:14.150 INFO:teuthology.orchestra.run.smithi122.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T11:50:14.150 INFO:teuthology.orchestra.run.smithi122.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T11:50:14.150 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T11:50:14.150 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 508 k 2024-09-13T11:50:14.150 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T11:50:14.150 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T11:50:14.150 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T11:50:14.150 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T11:50:14.150 INFO:teuthology.orchestra.run.smithi122.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T11:50:14.150 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 45 k 2024-09-13T11:50:14.150 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 143 k 2024-09-13T11:50:14.151 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T11:50:14.151 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T11:50:14.151 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 321 k 2024-09-13T11:50:14.151 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 305 k 2024-09-13T11:50:14.151 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 99 k 2024-09-13T11:50:14.151 INFO:teuthology.orchestra.run.smithi122.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-13T11:50:14.151 INFO:teuthology.orchestra.run.smithi122.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-13T11:50:14.151 INFO:teuthology.orchestra.run.smithi122.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T11:50:14.151 INFO:teuthology.orchestra.run.smithi122.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T11:50:14.151 INFO:teuthology.orchestra.run.smithi122.stdout:Installing weak dependencies: 2024-09-13T11:50:14.151 INFO:teuthology.orchestra.run.smithi122.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-13T11:50:14.151 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:50:14.151 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T11:50:14.152 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:50:14.152 INFO:teuthology.orchestra.run.smithi122.stdout:Install 30 Packages 2024-09-13T11:50:14.152 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrade 2 Packages 2024-09-13T11:50:14.152 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:50:14.152 INFO:teuthology.orchestra.run.smithi100.stdout:(9/32): python3-cephfs-19.3.0-4697.gd58f7306.el 1.6 MB/s | 161 kB 00:00 2024-09-13T11:50:14.153 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 62 M 2024-09-13T11:50:14.153 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T11:50:14.185 INFO:teuthology.orchestra.run.smithi100.stdout:(10/32): python3-rados-19.3.0-4697.gd58f7306.el 5.3 MB/s | 321 kB 00:00 2024-09-13T11:50:14.219 INFO:teuthology.orchestra.run.smithi100.stdout:(11/32): python3-rbd-19.3.0-4697.gd58f7306.el9. 4.4 MB/s | 305 kB 00:00 2024-09-13T11:50:14.244 INFO:teuthology.orchestra.run.smithi100.stdout:(12/32): python3-rgw-19.3.0-4697.gd58f7306.el9. 1.7 MB/s | 99 kB 00:00 2024-09-13T11:50:14.486 INFO:teuthology.orchestra.run.smithi100.stdout:(13/32): ceph-common-19.3.0-4697.gd58f7306.el9. 15 MB/s | 22 MB 00:01 2024-09-13T11:50:14.537 INFO:teuthology.orchestra.run.smithi100.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 337 kB/s | 107 kB 00:00 2024-09-13T11:50:14.629 INFO:teuthology.orchestra.run.smithi100.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 417 kB/s | 160 kB 00:00 2024-09-13T11:50:14.654 INFO:teuthology.orchestra.run.smithi100.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 271 kB/s | 45 kB 00:00 2024-09-13T11:50:14.712 INFO:teuthology.orchestra.run.smithi100.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 662 kB 00:00 2024-09-13T11:50:14.738 INFO:teuthology.orchestra.run.smithi100.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 505 kB/s | 42 kB 00:00 2024-09-13T11:50:14.763 INFO:teuthology.orchestra.run.smithi100.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-09-13T11:50:14.788 INFO:teuthology.orchestra.run.smithi100.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-09-13T11:50:14.813 INFO:teuthology.orchestra.run.smithi100.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 477 kB/s | 36 kB 00:00 2024-09-13T11:50:14.839 INFO:teuthology.orchestra.run.smithi100.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 628 kB/s | 47 kB 00:00 2024-09-13T11:50:14.898 INFO:teuthology.orchestra.run.smithi100.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-13T11:50:14.914 INFO:teuthology.orchestra.run.smithi100.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 3.2 MB/s | 49 kB 00:00 2024-09-13T11:50:14.941 INFO:teuthology.orchestra.run.smithi100.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 302 kB/s | 26 kB 00:00 2024-09-13T11:50:14.965 INFO:teuthology.orchestra.run.smithi100.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-13T11:50:15.001 INFO:teuthology.orchestra.run.smithi100.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.7 MB/s | 191 kB 00:00 2024-09-13T11:50:15.040 INFO:teuthology.orchestra.run.smithi100.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-09-13T11:50:15.099 INFO:teuthology.orchestra.run.smithi100.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-13T11:50:15.174 INFO:teuthology.orchestra.run.smithi100.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-13T11:50:15.708 INFO:teuthology.orchestra.run.smithi100.stdout:(31/32): librbd1-19.3.0-4697.gd58f7306.el9.x86_ 5.3 MB/s | 3.2 MB 00:00 2024-09-13T11:50:15.825 INFO:teuthology.orchestra.run.smithi100.stdout:(32/32): librados2-19.3.0-4697.gd58f7306.el9.x8 4.4 MB/s | 3.5 MB 00:00 2024-09-13T11:50:15.830 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:50:15.830 INFO:teuthology.orchestra.run.smithi100.stdout:Total 20 MB/s | 62 MB 00:03 2024-09-13T11:50:15.935 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T11:50:15.971 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T11:50:15.971 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T11:50:16.190 INFO:teuthology.orchestra.run.smithi122.stdout:(1/32): ceph-base-19.3.0-4697.gd58f7306.el9.x86 2.9 MB/s | 5.5 MB 00:01 2024-09-13T11:50:16.249 INFO:teuthology.orchestra.run.smithi122.stdout:(2/32): ceph-selinux-19.3.0-4697.gd58f7306.el9. 434 kB/s | 25 kB 00:00 2024-09-13T11:50:16.349 INFO:teuthology.orchestra.run.smithi122.stdout:(3/32): libcephfs2-19.3.0-4697.gd58f7306.el9.x8 7.4 MB/s | 755 kB 00:00 2024-09-13T11:50:16.416 INFO:teuthology.orchestra.run.smithi122.stdout:(4/32): libradosstriper1-19.3.0-4697.gd58f7306. 7.5 MB/s | 508 kB 00:00 2024-09-13T11:50:16.645 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T11:50:16.646 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T11:50:16.783 INFO:teuthology.orchestra.run.smithi122.stdout:(5/32): librgw2-19.3.0-4697.gd58f7306.el9.x86_6 15 MB/s | 5.5 MB 00:00 2024-09-13T11:50:16.825 INFO:teuthology.orchestra.run.smithi122.stdout:(6/32): python3-ceph-argparse-19.3.0-4697.gd58f 1.1 MB/s | 45 kB 00:00 2024-09-13T11:50:16.892 INFO:teuthology.orchestra.run.smithi122.stdout:(7/32): python3-ceph-common-19.3.0-4697.gd58f73 2.1 MB/s | 143 kB 00:00 2024-09-13T11:50:16.959 INFO:teuthology.orchestra.run.smithi122.stdout:(8/32): python3-cephfs-19.3.0-4697.gd58f7306.el 2.4 MB/s | 161 kB 00:00 2024-09-13T11:50:17.027 INFO:teuthology.orchestra.run.smithi122.stdout:(9/32): python3-rados-19.3.0-4697.gd58f7306.el9 4.6 MB/s | 321 kB 00:00 2024-09-13T11:50:17.076 INFO:teuthology.orchestra.run.smithi122.stdout:(10/32): python3-rbd-19.3.0-4697.gd58f7306.el9. 6.1 MB/s | 305 kB 00:00 2024-09-13T11:50:17.126 INFO:teuthology.orchestra.run.smithi122.stdout:(11/32): python3-rgw-19.3.0-4697.gd58f7306.el9. 1.9 MB/s | 99 kB 00:00 2024-09-13T11:50:17.268 INFO:teuthology.orchestra.run.smithi122.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 753 kB/s | 107 kB 00:00 2024-09-13T11:50:17.343 INFO:teuthology.orchestra.run.smithi122.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 160 kB 00:00 2024-09-13T11:50:17.385 INFO:teuthology.orchestra.run.smithi122.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-09-13T11:50:17.686 INFO:teuthology.orchestra.run.smithi122.stdout:(15/32): ceph-radosgw-19.3.0-4697.gd58f7306.el9 3.2 MB/s | 11 MB 00:03 2024-09-13T11:50:17.711 INFO:teuthology.orchestra.run.smithi122.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.0 MB/s | 662 kB 00:00 2024-09-13T11:50:17.770 INFO:teuthology.orchestra.run.smithi122.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 725 kB/s | 42 kB 00:00 2024-09-13T11:50:17.878 INFO:teuthology.orchestra.run.smithi122.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-09-13T11:50:17.904 INFO:teuthology.orchestra.run.smithi122.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.3 MB/s | 292 kB 00:00 2024-09-13T11:50:17.929 INFO:teuthology.orchestra.run.smithi122.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 713 kB/s | 36 kB 00:00 2024-09-13T11:50:17.963 INFO:teuthology.orchestra.run.smithi122.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 807 kB/s | 47 kB 00:00 2024-09-13T11:50:18.005 INFO:teuthology.orchestra.run.smithi122.stdout:(22/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 4.0 MB/s | 308 kB 00:00 2024-09-13T11:50:18.030 INFO:teuthology.orchestra.run.smithi122.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-13T11:50:18.063 INFO:teuthology.orchestra.run.smithi122.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-09-13T11:50:18.097 INFO:teuthology.orchestra.run.smithi122.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-09-13T11:50:18.143 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T11:50:18.172 INFO:teuthology.orchestra.run.smithi122.stdout:(26/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 11 MB/s | 839 kB 00:00 2024-09-13T11:50:18.206 INFO:teuthology.orchestra.run.smithi122.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-13T11:50:18.246 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-13T11:50:18.273 INFO:teuthology.orchestra.run.smithi122.stdout:(28/32): libarrow-9.0.0-11.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-09-13T11:50:18.280 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-13T11:50:18.305 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-13T11:50:18.323 INFO:teuthology.orchestra.run.smithi122.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-13T11:50:18.328 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T11:50:18.356 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T11:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T11:50:18.582 INFO:teuthology.orchestra.run.smithi122.stdout:(30/32): librados2-19.3.0-4697.gd58f7306.el9.x8 11 MB/s | 3.5 MB 00:00 2024-09-13T11:50:18.592 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T11:50:18.890 INFO:teuthology.orchestra.run.smithi122.stdout:(31/32): librbd1-19.3.0-4697.gd58f7306.el9.x86_ 5.7 MB/s | 3.2 MB 00:00 2024-09-13T11:50:18.932 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-13T11:50:19.048 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:50:19.138 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:50:19.244 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 8/34 2024-09-13T11:50:19.274 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T11:50:19.337 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T11:50:19.366 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-13T11:50:19.409 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 11/34 2024-09-13T11:50:19.478 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T11:50:19.510 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T11:50:19.574 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T11:50:19.609 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-13T11:50:19.730 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-13T11:50:19.760 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-13T11:50:19.809 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-13T11:50:19.833 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-13T11:50:19.854 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-13T11:50:19.879 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-13T11:50:20.197 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-13T11:50:20.250 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T11:50:20.341 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T11:50:20.375 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 23/34 2024-09-13T11:50:20.416 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-13T11:50:20.908 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-13T11:50:20.944 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T11:50:21.015 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T11:50:21.032 INFO:teuthology.orchestra.run.smithi122.stdout:(32/32): ceph-common-19.3.0-4697.gd58f7306.el9. 3.2 MB/s | 22 MB 00:06 2024-09-13T11:50:21.036 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:50:21.037 INFO:teuthology.orchestra.run.smithi122.stdout:Total 8.9 MB/s | 62 MB 00:06 2024-09-13T11:50:21.085 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 27/34 2024-09-13T11:50:21.113 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 28/34 2024-09-13T11:50:21.141 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T11:50:21.174 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T11:50:21.174 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T11:50:21.845 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T11:50:21.845 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T11:50:23.319 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T11:50:23.391 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-13T11:50:23.421 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-13T11:50:23.446 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-13T11:50:23.468 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T11:50:23.489 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T11:50:23.598 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:50:23.642 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:50:23.687 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T11:50:23.750 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T11:50:24.092 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-13T11:50:24.214 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:50:24.250 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:50:24.296 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T11:50:24.306 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:50:24.390 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T11:50:24.390 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T11:50:24.390 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T11:50:24.390 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:50:24.394 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 8/34 2024-09-13T11:50:24.424 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T11:50:24.430 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:50:24.487 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T11:50:24.515 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-13T11:50:24.559 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 11/34 2024-09-13T11:50:24.628 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T11:50:24.651 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T11:50:24.715 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T11:50:24.750 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-13T11:50:24.857 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-13T11:50:24.892 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-13T11:50:24.933 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-13T11:50:24.965 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-13T11:50:24.994 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-13T11:50:25.028 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-13T11:50:25.366 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-13T11:50:25.415 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T11:50:25.498 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T11:50:25.537 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 23/34 2024-09-13T11:50:25.581 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-13T11:50:26.073 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-13T11:50:26.101 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T11:50:26.173 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T11:50:26.253 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 27/34 2024-09-13T11:50:26.278 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 28/34 2024-09-13T11:50:28.797 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:50:28.833 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:50:29.414 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:50:29.453 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T11:50:29.541 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T11:50:29.541 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T11:50:29.541 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T11:50:29.541 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:50:29.579 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:50:34.457 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:50:34.457 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /sys 2024-09-13T11:50:34.457 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /proc 2024-09-13T11:50:34.457 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /mnt 2024-09-13T11:50:34.457 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /var/tmp 2024-09-13T11:50:34.457 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /home 2024-09-13T11:50:34.457 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /root 2024-09-13T11:50:34.457 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /tmp 2024-09-13T11:50:34.457 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:50:35.458 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T11:50:35.483 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T11:50:35.483 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:50:35.483 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T11:50:35.484 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:50:35.484 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:50:35.484 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:50:35.485 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T11:50:35.568 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T11:50:35.569 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:50:36.794 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:50:36.795 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 1/34 2024-09-13T11:50:36.795 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2/34 2024-09-13T11:50:36.795 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 3/34 2024-09-13T11:50:36.795 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 4/34 2024-09-13T11:50:36.795 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 5/34 2024-09-13T11:50:36.795 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 6/34 2024-09-13T11:50:36.795 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:50:36.795 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 8/34 2024-09-13T11:50:36.795 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 9/34 2024-09-13T11:50:36.795 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 10/34 2024-09-13T11:50:36.795 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 11/34 2024-09-13T11:50:36.796 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T11:50:36.796 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 13/34 2024-09-13T11:50:36.796 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-13T11:50:36.796 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-13T11:50:36.796 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-13T11:50:36.796 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-13T11:50:36.796 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-13T11:50:36.796 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-13T11:50:36.796 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-13T11:50:36.796 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-13T11:50:36.796 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-13T11:50:36.796 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-13T11:50:36.797 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-13T11:50:36.797 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-13T11:50:36.797 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-13T11:50:36.797 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-13T11:50:36.797 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-13T11:50:36.797 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-13T11:50:36.797 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-13T11:50:36.798 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:50:36.798 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-13T11:50:36.798 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 33/34 2024-09-13T11:50:37.784 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:50:37.784 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:50:37.784 INFO:teuthology.orchestra.run.smithi100.stdout:Upgraded: 2024-09-13T11:50:37.785 INFO:teuthology.orchestra.run.smithi100.stdout: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:37.785 INFO:teuthology.orchestra.run.smithi100.stdout: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:37.785 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T11:50:37.785 INFO:teuthology.orchestra.run.smithi100.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T11:50:37.785 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:37.785 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:37.785 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:37.785 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:37.785 INFO:teuthology.orchestra.run.smithi100.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T11:50:37.785 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T11:50:37.785 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T11:50:37.785 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:37.785 INFO:teuthology.orchestra.run.smithi100.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T11:50:37.785 INFO:teuthology.orchestra.run.smithi100.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T11:50:37.786 INFO:teuthology.orchestra.run.smithi100.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T11:50:37.786 INFO:teuthology.orchestra.run.smithi100.stdout: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:37.786 INFO:teuthology.orchestra.run.smithi100.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T11:50:37.786 INFO:teuthology.orchestra.run.smithi100.stdout: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:37.786 INFO:teuthology.orchestra.run.smithi100.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T11:50:37.786 INFO:teuthology.orchestra.run.smithi100.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T11:50:37.786 INFO:teuthology.orchestra.run.smithi100.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T11:50:37.786 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:37.786 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:37.786 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:37.786 INFO:teuthology.orchestra.run.smithi100.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T11:50:37.786 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:37.786 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:37.786 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:37.787 INFO:teuthology.orchestra.run.smithi100.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-13T11:50:37.787 INFO:teuthology.orchestra.run.smithi100.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-13T11:50:37.787 INFO:teuthology.orchestra.run.smithi100.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-13T11:50:37.787 INFO:teuthology.orchestra.run.smithi100.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T11:50:37.787 INFO:teuthology.orchestra.run.smithi100.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T11:50:37.787 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:50:37.787 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:50:38.122 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-test 2024-09-13T11:50:38.684 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:28 ago on Fri 13 Sep 2024 11:50:10 AM UTC. 2024-09-13T11:50:38.822 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:50:38.823 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:50:38.823 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-13T11:50:38.823 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:50:38.823 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T11:50:38.823 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-test x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 50 M 2024-09-13T11:50:38.823 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-13T11:50:38.823 INFO:teuthology.orchestra.run.smithi100.stdout: libcephsqlite x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T11:50:38.823 INFO:teuthology.orchestra.run.smithi100.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T11:50:38.823 INFO:teuthology.orchestra.run.smithi100.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T11:50:38.823 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:50:38.824 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T11:50:38.824 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:50:38.824 INFO:teuthology.orchestra.run.smithi100.stdout:Install 4 Packages 2024-09-13T11:50:38.824 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:50:38.824 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 51 M 2024-09-13T11:50:38.824 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 216 M 2024-09-13T11:50:38.824 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T11:50:39.024 INFO:teuthology.orchestra.run.smithi100.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.0 MB/s | 304 kB 00:00 2024-09-13T11:50:39.057 INFO:teuthology.orchestra.run.smithi100.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2024-09-13T11:50:39.107 INFO:teuthology.orchestra.run.smithi100.stdout:(3/4): libcephsqlite-19.3.0-4697.gd58f7306.el9. 692 kB/s | 161 kB 00:00 2024-09-13T11:50:39.474 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:50:39.474 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /sys 2024-09-13T11:50:39.474 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /proc 2024-09-13T11:50:39.474 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /mnt 2024-09-13T11:50:39.474 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /var/tmp 2024-09-13T11:50:39.474 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /home 2024-09-13T11:50:39.474 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /root 2024-09-13T11:50:39.474 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /tmp 2024-09-13T11:50:39.474 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:50:40.391 INFO:teuthology.orchestra.run.smithi100.stdout:(4/4): ceph-test-19.3.0-4697.gd58f7306.el9.x86_ 33 MB/s | 50 MB 00:01 2024-09-13T11:50:40.393 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:50:40.393 INFO:teuthology.orchestra.run.smithi100.stdout:Total 32 MB/s | 51 MB 00:01 2024-09-13T11:50:40.421 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T11:50:40.441 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T11:50:40.441 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T11:50:40.483 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T11:50:40.508 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T11:50:40.508 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:50:40.508 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T11:50:40.508 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:50:40.508 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:50:40.508 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:50:40.510 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T11:50:40.593 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T11:50:40.593 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:50:40.821 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T11:50:40.821 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T11:50:41.349 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T11:50:41.432 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T11:50:41.475 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T11:50:41.526 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T11:50:41.802 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:50:41.802 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 1/34 2024-09-13T11:50:41.802 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2/34 2024-09-13T11:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 3/34 2024-09-13T11:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 4/34 2024-09-13T11:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 5/34 2024-09-13T11:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 6/34 2024-09-13T11:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 8/34 2024-09-13T11:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 9/34 2024-09-13T11:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 10/34 2024-09-13T11:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 11/34 2024-09-13T11:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T11:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 13/34 2024-09-13T11:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-13T11:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-13T11:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-13T11:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-13T11:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-13T11:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-13T11:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-13T11:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-13T11:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-13T11:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-13T11:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-13T11:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-13T11:50:41.805 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-13T11:50:41.805 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-13T11:50:41.805 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-13T11:50:41.805 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-13T11:50:41.805 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-13T11:50:41.805 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:50:41.805 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-13T11:50:41.805 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 33/34 2024-09-13T11:50:42.885 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:50:42.885 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:50:42.885 INFO:teuthology.orchestra.run.smithi122.stdout:Upgraded: 2024-09-13T11:50:42.885 INFO:teuthology.orchestra.run.smithi122.stdout: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:42.885 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:42.885 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T11:50:42.886 INFO:teuthology.orchestra.run.smithi122.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T11:50:42.886 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:42.886 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:42.886 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:42.886 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:42.886 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T11:50:42.886 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T11:50:42.886 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T11:50:42.886 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:42.886 INFO:teuthology.orchestra.run.smithi122.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T11:50:42.886 INFO:teuthology.orchestra.run.smithi122.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T11:50:42.886 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T11:50:42.887 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:42.887 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T11:50:42.887 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:42.887 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T11:50:42.887 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T11:50:42.887 INFO:teuthology.orchestra.run.smithi122.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T11:50:42.887 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:42.887 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:42.887 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:42.887 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T11:50:42.887 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:42.887 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:42.887 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:42.887 INFO:teuthology.orchestra.run.smithi122.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-13T11:50:42.888 INFO:teuthology.orchestra.run.smithi122.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-13T11:50:42.888 INFO:teuthology.orchestra.run.smithi122.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-13T11:50:42.888 INFO:teuthology.orchestra.run.smithi122.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T11:50:42.888 INFO:teuthology.orchestra.run.smithi122.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T11:50:42.888 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:50:42.888 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:50:43.213 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-test 2024-09-13T11:50:43.776 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:32 ago on Fri 13 Sep 2024 11:50:11 AM UTC. 2024-09-13T11:50:43.909 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:50:43.909 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:50:43.909 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T11:50:43.909 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:50:43.910 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T11:50:43.910 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 50 M 2024-09-13T11:50:43.910 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T11:50:43.910 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T11:50:43.910 INFO:teuthology.orchestra.run.smithi122.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T11:50:43.910 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T11:50:43.910 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:50:43.910 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T11:50:43.910 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:50:43.910 INFO:teuthology.orchestra.run.smithi122.stdout:Install 4 Packages 2024-09-13T11:50:43.910 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:50:43.911 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 51 M 2024-09-13T11:50:43.911 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 216 M 2024-09-13T11:50:43.911 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T11:50:44.248 INFO:teuthology.orchestra.run.smithi122.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-09-13T11:50:44.432 INFO:teuthology.orchestra.run.smithi122.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 347 kB/s | 64 kB 00:00 2024-09-13T11:50:44.465 INFO:teuthology.orchestra.run.smithi122.stdout:(3/4): libcephsqlite-19.3.0-4697.gd58f7306.el9. 378 kB/s | 161 kB 00:00 2024-09-13T11:50:46.268 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T11:50:46.341 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T11:50:48.282 INFO:teuthology.orchestra.run.smithi122.stdout:(4/4): ceph-test-19.3.0-4697.gd58f7306.el9.x86_ 12 MB/s | 50 MB 00:04 2024-09-13T11:50:48.284 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:50:48.284 INFO:teuthology.orchestra.run.smithi122.stdout:Total 12 MB/s | 51 MB 00:04 2024-09-13T11:50:48.311 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T11:50:48.331 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T11:50:48.331 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T11:50:48.714 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T11:50:48.714 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T11:50:48.765 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T11:50:48.765 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 1/4 2024-09-13T11:50:48.766 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2/4 2024-09-13T11:50:48.766 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T11:50:49.063 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T11:50:49.063 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:50:49.063 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T11:50:49.063 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:49.064 INFO:teuthology.orchestra.run.smithi100.stdout: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:49.064 INFO:teuthology.orchestra.run.smithi100.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T11:50:49.064 INFO:teuthology.orchestra.run.smithi100.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T11:50:49.064 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:50:49.064 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:50:49.259 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T11:50:49.341 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph 2024-09-13T11:50:49.351 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T11:50:49.400 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T11:50:49.426 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T11:50:49.908 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:39 ago on Fri 13 Sep 2024 11:50:10 AM UTC. 2024-09-13T11:50:50.044 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:50:50.046 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:50:50.046 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-13T11:50:50.046 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:50:50.046 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T11:50:50.046 INFO:teuthology.orchestra.run.smithi100.stdout: ceph x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 6.6 k 2024-09-13T11:50:50.046 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-13T11:50:50.047 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mds x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 2.4 M 2024-09-13T11:50:50.047 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 1.5 M 2024-09-13T11:50:50.047 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 291 k 2024-09-13T11:50:50.047 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mon x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 4.8 M 2024-09-13T11:50:50.047 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-osd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 17 M 2024-09-13T11:50:50.047 INFO:teuthology.orchestra.run.smithi100.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-13T11:50:50.047 INFO:teuthology.orchestra.run.smithi100.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-13T11:50:50.047 INFO:teuthology.orchestra.run.smithi100.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-13T11:50:50.047 INFO:teuthology.orchestra.run.smithi100.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T11:50:50.047 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T11:50:50.047 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T11:50:50.047 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T11:50:50.047 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T11:50:50.048 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T11:50:50.048 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T11:50:50.048 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-collections 2024-09-13T11:50:50.048 INFO:teuthology.orchestra.run.smithi100.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-13T11:50:50.048 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T11:50:50.048 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T11:50:50.048 INFO:teuthology.orchestra.run.smithi100.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T11:50:50.048 INFO:teuthology.orchestra.run.smithi100.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T11:50:50.048 INFO:teuthology.orchestra.run.smithi100.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T11:50:50.048 INFO:teuthology.orchestra.run.smithi100.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T11:50:50.048 INFO:teuthology.orchestra.run.smithi100.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T11:50:50.048 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T11:50:50.048 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T11:50:50.049 INFO:teuthology.orchestra.run.smithi100.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T11:50:50.049 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T11:50:50.049 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T11:50:50.049 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T11:50:50.049 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T11:50:50.049 INFO:teuthology.orchestra.run.smithi100.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T11:50:50.049 INFO:teuthology.orchestra.run.smithi100.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T11:50:50.049 INFO:teuthology.orchestra.run.smithi100.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T11:50:50.049 INFO:teuthology.orchestra.run.smithi100.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T11:50:50.049 INFO:teuthology.orchestra.run.smithi100.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T11:50:50.049 INFO:teuthology.orchestra.run.smithi100.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T11:50:50.049 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:50:50.049 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T11:50:50.049 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:50:50.050 INFO:teuthology.orchestra.run.smithi100.stdout:Install 37 Packages 2024-09-13T11:50:50.050 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:50:50.050 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 31 M 2024-09-13T11:50:50.050 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 109 M 2024-09-13T11:50:50.050 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T11:50:50.675 INFO:teuthology.orchestra.run.smithi100.stdout:(1/37): ceph-19.3.0-4697.gd58f7306.el9.x86_64.r 48 kB/s | 6.6 kB 00:00 2024-09-13T11:50:50.842 INFO:teuthology.orchestra.run.smithi100.stdout:(2/37): ceph-mgr-19.3.0-4697.gd58f7306.el9.x86_ 5.0 MB/s | 1.5 MB 00:00 2024-09-13T11:50:50.917 INFO:teuthology.orchestra.run.smithi100.stdout:(3/37): ceph-mds-19.3.0-4697.gd58f7306.el9.x86_ 6.2 MB/s | 2.4 MB 00:00 2024-09-13T11:50:50.976 INFO:teuthology.orchestra.run.smithi100.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4697.gd58f 4.9 MB/s | 291 kB 00:00 2024-09-13T11:50:51.068 INFO:teuthology.orchestra.run.smithi100.stdout:(5/37): ceph-mon-19.3.0-4697.gd58f7306.el9.x86_ 12 MB/s | 4.8 MB 00:00 2024-09-13T11:50:51.143 INFO:teuthology.orchestra.run.smithi100.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-09-13T11:50:51.218 INFO:teuthology.orchestra.run.smithi100.stdout:(7/37): python3-cryptography-36.0.1-4.el9.x86_6 8.2 MB/s | 1.2 MB 00:00 2024-09-13T11:50:51.243 INFO:teuthology.orchestra.run.smithi100.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 106 kB 00:00 2024-09-13T11:50:51.294 INFO:teuthology.orchestra.run.smithi100.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 1.8 MB/s | 135 kB 00:00 2024-09-13T11:50:51.336 INFO:teuthology.orchestra.run.smithi100.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.3 MB/s | 125 kB 00:00 2024-09-13T11:50:51.361 INFO:teuthology.orchestra.run.smithi100.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-09-13T11:50:51.536 INFO:teuthology.orchestra.run.smithi100.stdout:(12/37): ceph-osd-19.3.0-4697.gd58f7306.el9.x86 25 MB/s | 17 MB 00:00 2024-09-13T11:50:51.603 INFO:teuthology.orchestra.run.smithi100.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 706 kB/s | 188 kB 00:00 2024-09-13T11:50:51.628 INFO:teuthology.orchestra.run.smithi100.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 379 kB/s | 35 kB 00:00 2024-09-13T11:50:51.654 INFO:teuthology.orchestra.run.smithi100.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-13T11:50:51.679 INFO:teuthology.orchestra.run.smithi100.stdout:(16/37): python3-mako-1.1.4-6.el9.noarch.rpm 542 kB/s | 172 kB 00:00 2024-09-13T11:50:51.704 INFO:teuthology.orchestra.run.smithi100.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 676 kB/s | 51 kB 00:00 2024-09-13T11:50:51.730 INFO:teuthology.orchestra.run.smithi100.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 551 kB/s | 42 kB 00:00 2024-09-13T11:50:51.780 INFO:teuthology.orchestra.run.smithi100.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-13T11:50:51.805 INFO:teuthology.orchestra.run.smithi100.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 578 kB/s | 43 kB 00:00 2024-09-13T11:50:51.830 INFO:teuthology.orchestra.run.smithi100.stdout:(21/37): lua-devel-5.4.4-4.el9.x86_64.rpm 147 kB/s | 22 kB 00:00 2024-09-13T11:50:51.864 INFO:teuthology.orchestra.run.smithi100.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-13T11:50:51.889 INFO:teuthology.orchestra.run.smithi100.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 182 kB/s | 11 kB 00:00 2024-09-13T11:50:51.914 INFO:teuthology.orchestra.run.smithi100.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 364 kB/s | 18 kB 00:00 2024-09-13T11:50:51.940 INFO:teuthology.orchestra.run.smithi100.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 459 kB/s | 23 kB 00:00 2024-09-13T11:50:51.965 INFO:teuthology.orchestra.run.smithi100.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-09-13T11:50:52.018 INFO:teuthology.orchestra.run.smithi100.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 251 kB/s | 20 kB 00:00 2024-09-13T11:50:52.049 INFO:teuthology.orchestra.run.smithi100.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 554 kB/s | 46 kB 00:00 2024-09-13T11:50:52.074 INFO:teuthology.orchestra.run.smithi100.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-13T11:50:52.099 INFO:teuthology.orchestra.run.smithi100.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 977 kB/s | 79 kB 00:00 2024-09-13T11:50:52.125 INFO:teuthology.orchestra.run.smithi100.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-13T11:50:52.150 INFO:teuthology.orchestra.run.smithi100.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-13T11:50:52.176 INFO:teuthology.orchestra.run.smithi100.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 710 kB/s | 36 kB 00:00 2024-09-13T11:50:52.201 INFO:teuthology.orchestra.run.smithi100.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-13T11:50:52.226 INFO:teuthology.orchestra.run.smithi100.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 797 kB/s | 20 kB 00:00 2024-09-13T11:50:52.251 INFO:teuthology.orchestra.run.smithi100.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-13T11:50:52.276 INFO:teuthology.orchestra.run.smithi100.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-13T11:50:52.280 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:50:52.280 INFO:teuthology.orchestra.run.smithi100.stdout:Total 14 MB/s | 31 MB 00:02 2024-09-13T11:50:52.427 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T11:50:52.467 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T11:50:52.467 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T11:50:52.944 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T11:50:52.944 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T11:50:54.320 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T11:50:54.352 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T11:50:54.408 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T11:50:54.416 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-13T11:50:54.448 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-13T11:50:54.477 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-13T11:50:54.548 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-13T11:50:54.601 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-13T11:50:54.633 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-13T11:50:54.724 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-13T11:50:54.803 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-13T11:50:54.844 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-13T11:50:54.879 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-13T11:50:54.909 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-13T11:50:54.944 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-13T11:50:54.977 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-13T11:50:55.026 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-13T11:50:55.061 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-13T11:50:55.095 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-13T11:50:55.143 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-13T11:50:55.177 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-13T11:50:55.248 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-13T11:50:55.386 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-13T11:50:55.475 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-13T11:50:55.529 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-13T11:50:55.567 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-13T11:50:55.616 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-13T11:50:55.679 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-13T11:50:55.863 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-13T11:50:55.917 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-13T11:50:55.993 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-13T11:50:56.081 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-13T11:50:56.123 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-13T11:50:56.250 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-13T11:50:56.412 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 32/37 2024-09-13T11:50:56.441 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T11:50:56.467 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T11:50:56.467 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:50:56.467 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T11:50:56.467 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T11:50:56.467 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T11:50:56.467 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:50:57.052 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T11:50:57.052 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 1/4 2024-09-13T11:50:57.052 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2/4 2024-09-13T11:50:57.052 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T11:50:57.357 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T11:50:57.358 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:50:57.358 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T11:50:57.358 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:57.358 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:50:57.358 INFO:teuthology.orchestra.run.smithi122.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T11:50:57.358 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T11:50:57.358 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:50:57.358 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:50:57.599 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph 2024-09-13T11:50:57.934 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T11:50:57.964 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T11:50:57.964 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:50:57.964 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T11:50:57.964 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T11:50:57.964 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T11:50:57.965 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:50:58.164 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:47 ago on Fri 13 Sep 2024 11:50:11 AM UTC. 2024-09-13T11:50:58.299 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:50:58.301 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:50:58.301 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T11:50:58.301 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:50:58.301 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T11:50:58.301 INFO:teuthology.orchestra.run.smithi122.stdout: ceph x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 6.6 k 2024-09-13T11:50:58.302 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T11:50:58.302 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 2.4 M 2024-09-13T11:50:58.302 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 1.5 M 2024-09-13T11:50:58.302 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 291 k 2024-09-13T11:50:58.302 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 4.8 M 2024-09-13T11:50:58.302 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 17 M 2024-09-13T11:50:58.302 INFO:teuthology.orchestra.run.smithi122.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-13T11:50:58.302 INFO:teuthology.orchestra.run.smithi122.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-13T11:50:58.302 INFO:teuthology.orchestra.run.smithi122.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-13T11:50:58.302 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T11:50:58.302 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T11:50:58.302 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T11:50:58.302 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T11:50:58.303 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T11:50:58.303 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T11:50:58.303 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T11:50:58.303 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-collections 2024-09-13T11:50:58.303 INFO:teuthology.orchestra.run.smithi122.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-13T11:50:58.303 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T11:50:58.303 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T11:50:58.303 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T11:50:58.303 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T11:50:58.303 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T11:50:58.303 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T11:50:58.303 INFO:teuthology.orchestra.run.smithi122.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T11:50:58.304 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T11:50:58.304 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T11:50:58.304 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T11:50:58.304 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T11:50:58.304 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T11:50:58.304 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T11:50:58.304 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T11:50:58.304 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T11:50:58.304 INFO:teuthology.orchestra.run.smithi122.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T11:50:58.304 INFO:teuthology.orchestra.run.smithi122.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T11:50:58.304 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T11:50:58.304 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T11:50:58.304 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T11:50:58.305 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:50:58.305 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T11:50:58.305 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:50:58.305 INFO:teuthology.orchestra.run.smithi122.stdout:Install 37 Packages 2024-09-13T11:50:58.305 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:50:58.305 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 31 M 2024-09-13T11:50:58.305 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 109 M 2024-09-13T11:50:58.306 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T11:50:58.410 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T11:50:58.434 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T11:50:58.434 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:50:58.434 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T11:50:58.434 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T11:50:58.434 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T11:50:58.434 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:50:58.669 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T11:50:58.693 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T11:50:58.693 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:50:58.693 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T11:50:58.693 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T11:50:58.693 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T11:50:58.693 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:50:58.757 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T11:50:58.958 INFO:teuthology.orchestra.run.smithi122.stdout:(1/37): ceph-19.3.0-4697.gd58f7306.el9.x86_64.r 48 kB/s | 6.6 kB 00:00 2024-09-13T11:50:59.167 INFO:teuthology.orchestra.run.smithi122.stdout:(2/37): ceph-mgr-19.3.0-4697.gd58f7306.el9.x86_ 4.4 MB/s | 1.5 MB 00:00 2024-09-13T11:50:59.617 INFO:teuthology.orchestra.run.smithi122.stdout:(3/37): ceph-mds-19.3.0-4697.gd58f7306.el9.x86_ 3.0 MB/s | 2.4 MB 00:00 2024-09-13T11:50:59.701 INFO:teuthology.orchestra.run.smithi122.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4697.gd58f 3.4 MB/s | 291 kB 00:00 2024-09-13T11:50:59.976 INFO:teuthology.orchestra.run.smithi122.stdout:(5/37): ceph-osd-19.3.0-4697.gd58f7306.el9.x86_ 21 MB/s | 17 MB 00:00 2024-09-13T11:51:00.001 INFO:teuthology.orchestra.run.smithi122.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 844 kB/s | 253 kB 00:00 2024-09-13T11:51:00.060 INFO:teuthology.orchestra.run.smithi122.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-09-13T11:51:00.118 INFO:teuthology.orchestra.run.smithi122.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-09-13T11:51:00.168 INFO:teuthology.orchestra.run.smithi122.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-09-13T11:51:00.227 INFO:teuthology.orchestra.run.smithi122.stdout:(10/37): ceph-mon-19.3.0-4697.gd58f7306.el9.x86 3.8 MB/s | 4.8 MB 00:01 2024-09-13T11:51:00.253 INFO:teuthology.orchestra.run.smithi122.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.5 MB/s | 215 kB 00:00 2024-09-13T11:51:00.295 INFO:teuthology.orchestra.run.smithi122.stdout:(12/37): python3-cryptography-36.0.1-4.el9.x86_ 3.9 MB/s | 1.2 MB 00:00 2024-09-13T11:51:00.420 INFO:teuthology.orchestra.run.smithi122.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 278 kB/s | 35 kB 00:00 2024-09-13T11:51:00.462 INFO:teuthology.orchestra.run.smithi122.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 805 kB/s | 188 kB 00:00 2024-09-13T11:51:00.487 INFO:teuthology.orchestra.run.smithi122.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 735 kB/s | 172 kB 00:00 2024-09-13T11:51:00.512 INFO:teuthology.orchestra.run.smithi122.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 837 kB/s | 77 kB 00:00 2024-09-13T11:51:00.538 INFO:teuthology.orchestra.run.smithi122.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 675 kB/s | 51 kB 00:00 2024-09-13T11:51:00.551 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T11:51:00.552 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 1/37 2024-09-13T11:51:00.552 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2/37 2024-09-13T11:51:00.552 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 3/37 2024-09-13T11:51:00.552 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 4/37 2024-09-13T11:51:00.552 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 5/37 2024-09-13T11:51:00.552 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 6/37 2024-09-13T11:51:00.552 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-13T11:51:00.553 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-13T11:51:00.553 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-13T11:51:00.553 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-13T11:51:00.553 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-13T11:51:00.553 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-13T11:51:00.553 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-13T11:51:00.553 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-13T11:51:00.553 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-13T11:51:00.553 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-13T11:51:00.553 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-13T11:51:00.554 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-13T11:51:00.554 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-13T11:51:00.554 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-13T11:51:00.554 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-13T11:51:00.554 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-13T11:51:00.555 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-13T11:51:00.555 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-13T11:51:00.555 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-13T11:51:00.555 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-13T11:51:00.555 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-13T11:51:00.555 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-13T11:51:00.555 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-13T11:51:00.555 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-13T11:51:00.555 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-13T11:51:00.556 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-13T11:51:00.556 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-13T11:51:00.556 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-13T11:51:00.556 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-13T11:51:00.556 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-13T11:51:00.563 INFO:teuthology.orchestra.run.smithi122.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 552 kB/s | 42 kB 00:00 2024-09-13T11:51:00.605 INFO:teuthology.orchestra.run.smithi122.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-13T11:51:00.630 INFO:teuthology.orchestra.run.smithi122.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 189 kB/s | 22 kB 00:00 2024-09-13T11:51:00.656 INFO:teuthology.orchestra.run.smithi122.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 470 kB/s | 43 kB 00:00 2024-09-13T11:51:00.681 INFO:teuthology.orchestra.run.smithi122.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 604 kB/s | 11 kB 00:00 2024-09-13T11:51:00.706 INFO:teuthology.orchestra.run.smithi122.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-09-13T11:51:00.823 INFO:teuthology.orchestra.run.smithi122.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 126 kB/s | 18 kB 00:00 2024-09-13T11:51:00.840 INFO:teuthology.orchestra.run.smithi122.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 173 kB/s | 23 kB 00:00 2024-09-13T11:51:00.857 INFO:teuthology.orchestra.run.smithi122.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 580 kB/s | 19 kB 00:00 2024-09-13T11:51:00.874 INFO:teuthology.orchestra.run.smithi122.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 588 kB/s | 20 kB 00:00 2024-09-13T11:51:00.891 INFO:teuthology.orchestra.run.smithi122.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2024-09-13T11:51:00.917 INFO:teuthology.orchestra.run.smithi122.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-09-13T11:51:00.933 INFO:teuthology.orchestra.run.smithi122.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 996 kB/s | 16 kB 00:00 2024-09-13T11:51:00.962 INFO:teuthology.orchestra.run.smithi122.stdout:(31/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.1 MB/s | 358 kB 00:00 2024-09-13T11:51:00.987 INFO:teuthology.orchestra.run.smithi122.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.6 MB/s | 90 kB 00:00 2024-09-13T11:51:01.012 INFO:teuthology.orchestra.run.smithi122.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-09-13T11:51:01.046 INFO:teuthology.orchestra.run.smithi122.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-13T11:51:01.071 INFO:teuthology.orchestra.run.smithi122.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 804 kB/s | 20 kB 00:00 2024-09-13T11:51:01.097 INFO:teuthology.orchestra.run.smithi122.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-09-13T11:51:01.122 INFO:teuthology.orchestra.run.smithi122.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-13T11:51:01.126 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:51:01.126 INFO:teuthology.orchestra.run.smithi122.stdout:Total 11 MB/s | 31 MB 00:02 2024-09-13T11:51:01.272 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T11:51:01.312 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T11:51:01.312 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T11:51:01.473 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-13T11:51:01.473 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:01.473 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T11:51:01.473 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:51:01.473 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:51:01.474 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:51:01.474 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:51:01.474 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:51:01.474 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:51:01.474 INFO:teuthology.orchestra.run.smithi100.stdout: lua-5.4.4-4.el9.x86_64 2024-09-13T11:51:01.474 INFO:teuthology.orchestra.run.smithi100.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-13T11:51:01.475 INFO:teuthology.orchestra.run.smithi100.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-13T11:51:01.475 INFO:teuthology.orchestra.run.smithi100.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T11:51:01.475 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T11:51:01.475 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T11:51:01.475 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T11:51:01.475 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T11:51:01.475 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T11:51:01.475 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T11:51:01.475 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T11:51:01.475 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T11:51:01.475 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T11:51:01.476 INFO:teuthology.orchestra.run.smithi100.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T11:51:01.476 INFO:teuthology.orchestra.run.smithi100.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T11:51:01.476 INFO:teuthology.orchestra.run.smithi100.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T11:51:01.476 INFO:teuthology.orchestra.run.smithi100.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T11:51:01.476 INFO:teuthology.orchestra.run.smithi100.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T11:51:01.476 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T11:51:01.476 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T11:51:01.477 INFO:teuthology.orchestra.run.smithi100.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T11:51:01.477 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T11:51:01.477 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T11:51:01.477 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T11:51:01.477 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T11:51:01.477 INFO:teuthology.orchestra.run.smithi100.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T11:51:01.477 INFO:teuthology.orchestra.run.smithi100.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T11:51:01.477 INFO:teuthology.orchestra.run.smithi100.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T11:51:01.477 INFO:teuthology.orchestra.run.smithi100.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T11:51:01.478 INFO:teuthology.orchestra.run.smithi100.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T11:51:01.478 INFO:teuthology.orchestra.run.smithi100.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T11:51:01.478 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:01.478 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:51:01.787 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T11:51:01.788 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T11:51:01.824 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-base 2024-09-13T11:51:02.392 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:52 ago on Fri 13 Sep 2024 11:50:10 AM UTC. 2024-09-13T11:51:02.482 INFO:teuthology.orchestra.run.smithi100.stdout:Package ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:51:02.524 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:51:02.525 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-13T11:51:02.525 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:51:02.591 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install cephadm 2024-09-13T11:51:03.151 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:53 ago on Fri 13 Sep 2024 11:50:10 AM UTC. 2024-09-13T11:51:03.161 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T11:51:03.241 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-13T11:51:03.274 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-13T11:51:03.281 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:51:03.282 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:03.282 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-13T11:51:03.282 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:03.282 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T11:51:03.282 INFO:teuthology.orchestra.run.smithi100.stdout: cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 783 k 2024-09-13T11:51:03.282 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:03.282 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T11:51:03.282 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:03.282 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-13T11:51:03.283 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:03.283 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 783 k 2024-09-13T11:51:03.283 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 789 k 2024-09-13T11:51:03.283 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T11:51:03.311 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-13T11:51:03.381 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-13T11:51:03.426 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-13T11:51:03.459 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-13T11:51:03.550 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-13T11:51:03.551 INFO:teuthology.orchestra.run.smithi100.stdout:cephadm-19.3.0-4697.gd58f7306.el9.noarch.rpm 2.9 MB/s | 783 kB 00:00 2024-09-13T11:51:03.551 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:51:03.552 INFO:teuthology.orchestra.run.smithi100.stdout:Total 2.8 MB/s | 783 kB 00:00 2024-09-13T11:51:03.552 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T11:51:03.557 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T11:51:03.557 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T11:51:03.563 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T11:51:03.563 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T11:51:03.637 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T11:51:03.665 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-13T11:51:03.712 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-13T11:51:03.747 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-13T11:51:03.785 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-13T11:51:03.828 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-13T11:51:03.862 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-13T11:51:03.902 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-13T11:51:03.937 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-13T11:51:03.972 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-13T11:51:04.004 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:51:04.019 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-13T11:51:04.061 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-13T11:51:04.095 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:51:04.140 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-13T11:51:04.287 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-13T11:51:04.367 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-13T11:51:04.421 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-13T11:51:04.461 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-13T11:51:04.486 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:51:04.510 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-13T11:51:04.565 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-13T11:51:04.719 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:51:04.719 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:04.719 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T11:51:04.719 INFO:teuthology.orchestra.run.smithi100.stdout: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:51:04.720 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:04.720 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:51:04.749 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-13T11:51:04.799 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-13T11:51:04.867 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-13T11:51:04.937 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T11:51:04.956 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-13T11:51:05.006 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-13T11:51:05.133 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-13T11:51:05.295 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 32/37 2024-09-13T11:51:05.325 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T11:51:05.351 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T11:51:05.351 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:51:05.351 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T11:51:05.351 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T11:51:05.351 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T11:51:05.351 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:05.503 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:55 ago on Fri 13 Sep 2024 11:50:10 AM UTC. 2024-09-13T11:51:05.636 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:51:05.636 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:05.636 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-09-13T11:51:05.636 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:05.637 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T11:51:05.637 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 145 k 2024-09-13T11:51:05.637 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:05.637 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T11:51:05.637 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:05.637 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-13T11:51:05.637 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:05.637 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 145 k 2024-09-13T11:51:05.637 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 432 k 2024-09-13T11:51:05.637 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T11:51:05.850 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-immutable-object-cache-19.3.0-4697.gd58f73 682 kB/s | 145 kB 00:00 2024-09-13T11:51:05.851 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:51:05.851 INFO:teuthology.orchestra.run.smithi100.stdout:Total 676 kB/s | 145 kB 00:00 2024-09-13T11:51:05.851 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T11:51:05.858 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T11:51:05.858 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T11:51:05.915 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T11:51:05.915 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T11:51:06.054 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T11:51:06.135 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:51:06.159 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:51:06.159 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:51:06.159 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T11:51:06.159 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:06.826 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T11:51:06.853 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T11:51:06.853 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:51:06.853 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T11:51:06.853 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T11:51:06.853 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T11:51:06.854 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:07.084 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:51:07.084 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:07.084 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T11:51:07.084 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:51:07.084 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:07.084 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:51:07.310 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T11:51:07.319 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr 2024-09-13T11:51:07.334 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T11:51:07.334 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:51:07.334 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T11:51:07.334 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T11:51:07.334 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T11:51:07.334 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:07.561 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T11:51:07.584 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T11:51:07.584 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:51:07.584 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T11:51:07.585 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T11:51:07.585 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T11:51:07.585 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:07.649 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T11:51:07.898 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:57 ago on Fri 13 Sep 2024 11:50:10 AM UTC. 2024-09-13T11:51:07.992 INFO:teuthology.orchestra.run.smithi100.stdout:Package ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:51:08.034 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:51:08.035 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-13T11:51:08.035 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:51:08.096 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T11:51:08.687 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:58 ago on Fri 13 Sep 2024 11:50:10 AM UTC. 2024-09-13T11:51:08.824 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:51:08.825 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:08.825 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-13T11:51:08.826 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:08.826 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T11:51:08.826 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 14 M 2024-09-13T11:51:08.826 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-13T11:51:08.826 INFO:teuthology.orchestra.run.smithi100.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-13T11:51:08.826 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 34 k 2024-09-13T11:51:08.826 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 16 k 2024-09-13T11:51:08.826 INFO:teuthology.orchestra.run.smithi100.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-13T11:51:08.826 INFO:teuthology.orchestra.run.smithi100.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-13T11:51:08.826 INFO:teuthology.orchestra.run.smithi100.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-13T11:51:08.827 INFO:teuthology.orchestra.run.smithi100.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-13T11:51:08.827 INFO:teuthology.orchestra.run.smithi100.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-13T11:51:08.827 INFO:teuthology.orchestra.run.smithi100.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-13T11:51:08.827 INFO:teuthology.orchestra.run.smithi100.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T11:51:08.827 INFO:teuthology.orchestra.run.smithi100.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T11:51:08.828 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:08.828 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T11:51:08.828 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:08.828 INFO:teuthology.orchestra.run.smithi100.stdout:Install 12 Packages 2024-09-13T11:51:08.828 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:08.829 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 19 M 2024-09-13T11:51:08.829 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 159 M 2024-09-13T11:51:08.829 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T11:51:09.331 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T11:51:09.332 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 1/37 2024-09-13T11:51:09.332 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2/37 2024-09-13T11:51:09.332 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 3/37 2024-09-13T11:51:09.332 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 4/37 2024-09-13T11:51:09.332 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 5/37 2024-09-13T11:51:09.332 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 6/37 2024-09-13T11:51:09.332 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-13T11:51:09.332 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-13T11:51:09.332 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-13T11:51:09.332 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-13T11:51:09.333 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-13T11:51:09.333 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-13T11:51:09.333 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-13T11:51:09.333 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-13T11:51:09.333 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-13T11:51:09.333 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-13T11:51:09.333 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-13T11:51:09.333 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-13T11:51:09.333 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-13T11:51:09.333 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-13T11:51:09.333 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-13T11:51:09.333 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-13T11:51:09.334 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-13T11:51:09.334 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-13T11:51:09.334 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-13T11:51:09.334 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-13T11:51:09.334 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-13T11:51:09.334 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-13T11:51:09.334 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-13T11:51:09.334 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-13T11:51:09.334 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-13T11:51:09.334 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-13T11:51:09.334 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-13T11:51:09.334 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-13T11:51:09.335 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-13T11:51:09.335 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-13T11:51:09.513 INFO:teuthology.orchestra.run.smithi100.stdout:(1/12): ceph-prometheus-alerts-19.3.0-4697.gd58 61 kB/s | 16 kB 00:00 2024-09-13T11:51:09.538 INFO:teuthology.orchestra.run.smithi100.stdout:(2/12): ceph-grafana-dashboards-19.3.0-4697.gd5 116 kB/s | 34 kB 00:00 2024-09-13T11:51:10.180 INFO:teuthology.orchestra.run.smithi100.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 419 kB/s | 269 kB 00:00 2024-09-13T11:51:10.229 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-13T11:51:10.229 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:10.229 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T11:51:10.229 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:51:10.229 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:51:10.229 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:51:10.229 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:51:10.229 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:51:10.229 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:51:10.229 INFO:teuthology.orchestra.run.smithi122.stdout: lua-5.4.4-4.el9.x86_64 2024-09-13T11:51:10.229 INFO:teuthology.orchestra.run.smithi122.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-13T11:51:10.230 INFO:teuthology.orchestra.run.smithi122.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-13T11:51:10.230 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T11:51:10.230 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T11:51:10.230 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T11:51:10.230 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T11:51:10.230 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T11:51:10.230 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T11:51:10.230 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T11:51:10.230 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T11:51:10.230 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T11:51:10.230 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T11:51:10.230 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T11:51:10.230 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T11:51:10.231 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T11:51:10.231 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T11:51:10.231 INFO:teuthology.orchestra.run.smithi122.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T11:51:10.231 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T11:51:10.231 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T11:51:10.231 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T11:51:10.231 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T11:51:10.231 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T11:51:10.231 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T11:51:10.231 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T11:51:10.231 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T11:51:10.231 INFO:teuthology.orchestra.run.smithi122.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T11:51:10.231 INFO:teuthology.orchestra.run.smithi122.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T11:51:10.231 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T11:51:10.232 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T11:51:10.232 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T11:51:10.232 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:10.232 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:51:10.356 INFO:teuthology.orchestra.run.smithi100.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.2 MB/s | 1.0 MB 00:00 2024-09-13T11:51:10.417 INFO:teuthology.orchestra.run.smithi100.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-13T11:51:10.431 INFO:teuthology.orchestra.run.smithi100.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-09-13T11:51:10.490 INFO:teuthology.orchestra.run.smithi100.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 35 MB/s | 2.0 MB 00:00 2024-09-13T11:51:10.523 INFO:teuthology.orchestra.run.smithi100.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.5 MB/s | 863 kB 00:00 2024-09-13T11:51:10.557 INFO:teuthology.orchestra.run.smithi100.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.1 MB/s | 144 kB 00:00 2024-09-13T11:51:10.582 INFO:teuthology.orchestra.run.smithi100.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 525 kB/s | 31 kB 00:00 2024-09-13T11:51:10.606 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-base 2024-09-13T11:51:10.607 INFO:teuthology.orchestra.run.smithi100.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.8 MB/s | 188 kB 00:00 2024-09-13T11:51:10.958 INFO:teuthology.orchestra.run.smithi100.stdout:(12/12): ceph-mgr-dashboard-19.3.0-4697.gd58f73 8.3 MB/s | 14 MB 00:01 2024-09-13T11:51:10.961 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:51:10.962 INFO:teuthology.orchestra.run.smithi100.stdout:Total 9.0 MB/s | 19 MB 00:02 2024-09-13T11:51:11.009 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T11:51:11.028 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T11:51:11.028 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T11:51:11.182 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:00 ago on Fri 13 Sep 2024 11:50:11 AM UTC. 2024-09-13T11:51:11.272 INFO:teuthology.orchestra.run.smithi122.stdout:Package ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:51:11.290 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T11:51:11.291 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T11:51:11.313 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:51:11.314 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T11:51:11.314 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:51:11.391 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install cephadm 2024-09-13T11:51:11.899 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T11:51:11.957 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:00 ago on Fri 13 Sep 2024 11:50:11 AM UTC. 2024-09-13T11:51:11.958 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-13T11:51:11.994 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-13T11:51:12.051 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-13T11:51:12.071 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-13T11:51:12.091 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:51:12.092 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:12.092 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T11:51:12.092 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:12.092 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T11:51:12.092 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 783 k 2024-09-13T11:51:12.092 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:12.092 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T11:51:12.092 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:12.092 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-13T11:51:12.092 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:12.092 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 783 k 2024-09-13T11:51:12.092 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 789 k 2024-09-13T11:51:12.093 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T11:51:12.174 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-13T11:51:12.347 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-13T11:51:12.354 INFO:teuthology.orchestra.run.smithi122.stdout:cephadm-19.3.0-4697.gd58f7306.el9.noarch.rpm 2.9 MB/s | 783 kB 00:00 2024-09-13T11:51:12.354 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:51:12.355 INFO:teuthology.orchestra.run.smithi122.stdout:Total 2.9 MB/s | 783 kB 00:00 2024-09-13T11:51:12.355 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T11:51:12.360 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T11:51:12.360 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T11:51:12.366 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T11:51:12.366 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T11:51:12.436 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T11:51:12.489 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-13T11:51:12.550 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-13T11:51:12.579 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-13T11:51:12.636 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 10/12 2024-09-13T11:51:12.907 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:51:13.007 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:51:13.382 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:51:13.637 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:51:13.638 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:13.638 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T11:51:13.638 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:51:13.638 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:13.638 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:51:13.858 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T11:51:14.425 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:03 ago on Fri 13 Sep 2024 11:50:11 AM UTC. 2024-09-13T11:51:14.557 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:51:14.558 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:14.558 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-09-13T11:51:14.558 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:14.558 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T11:51:14.558 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 145 k 2024-09-13T11:51:14.558 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:14.559 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T11:51:14.559 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:14.559 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-13T11:51:14.559 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:14.559 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 145 k 2024-09-13T11:51:14.559 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 432 k 2024-09-13T11:51:14.559 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T11:51:14.760 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-immutable-object-cache-19.3.0-4697.gd58f73 722 kB/s | 145 kB 00:00 2024-09-13T11:51:14.760 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:51:14.761 INFO:teuthology.orchestra.run.smithi122.stdout:Total 715 kB/s | 145 kB 00:00 2024-09-13T11:51:14.761 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T11:51:14.768 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T11:51:14.769 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T11:51:14.825 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T11:51:14.825 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T11:51:14.971 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T11:51:15.070 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:51:15.095 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:51:15.095 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:51:15.095 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T11:51:15.095 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:15.809 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 11/12 2024-09-13T11:51:15.866 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T11:51:16.005 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:51:16.005 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:16.006 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T11:51:16.006 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:51:16.006 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:16.006 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:51:16.253 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr 2024-09-13T11:51:16.821 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:05 ago on Fri 13 Sep 2024 11:50:11 AM UTC. 2024-09-13T11:51:16.914 INFO:teuthology.orchestra.run.smithi122.stdout:Package ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:51:16.957 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:51:16.958 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T11:51:16.959 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:51:17.029 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T11:51:17.583 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:06 ago on Fri 13 Sep 2024 11:50:11 AM UTC. 2024-09-13T11:51:17.704 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T11:51:17.704 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 1/12 2024-09-13T11:51:17.704 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 2/12 2024-09-13T11:51:17.704 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 3/12 2024-09-13T11:51:17.704 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-13T11:51:17.704 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-13T11:51:17.704 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-13T11:51:17.705 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-13T11:51:17.705 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-13T11:51:17.705 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-13T11:51:17.705 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-13T11:51:17.705 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-13T11:51:17.717 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:51:17.718 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:17.718 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T11:51:17.718 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:17.718 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T11:51:17.718 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 14 M 2024-09-13T11:51:17.718 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T11:51:17.719 INFO:teuthology.orchestra.run.smithi122.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-13T11:51:17.719 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 34 k 2024-09-13T11:51:17.719 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 16 k 2024-09-13T11:51:17.719 INFO:teuthology.orchestra.run.smithi122.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-13T11:51:17.719 INFO:teuthology.orchestra.run.smithi122.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-13T11:51:17.719 INFO:teuthology.orchestra.run.smithi122.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-13T11:51:17.719 INFO:teuthology.orchestra.run.smithi122.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-13T11:51:17.719 INFO:teuthology.orchestra.run.smithi122.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-13T11:51:17.719 INFO:teuthology.orchestra.run.smithi122.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-13T11:51:17.719 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T11:51:17.719 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T11:51:17.719 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:17.719 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T11:51:17.720 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:17.720 INFO:teuthology.orchestra.run.smithi122.stdout:Install 12 Packages 2024-09-13T11:51:17.720 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:17.720 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 19 M 2024-09-13T11:51:17.720 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 159 M 2024-09-13T11:51:17.720 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T11:51:18.035 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-13T11:51:18.035 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:18.035 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T11:51:18.035 INFO:teuthology.orchestra.run.smithi100.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-13T11:51:18.035 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:51:18.035 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:51:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:51:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-13T11:51:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-13T11:51:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-13T11:51:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-13T11:51:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-13T11:51:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-13T11:51:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T11:51:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T11:51:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:18.036 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:51:18.309 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T11:51:18.356 INFO:teuthology.orchestra.run.smithi122.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4697.gd5 237 kB/s | 34 kB 00:00 2024-09-13T11:51:18.381 INFO:teuthology.orchestra.run.smithi122.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4697.gd58 97 kB/s | 16 kB 00:00 2024-09-13T11:51:18.548 INFO:teuthology.orchestra.run.smithi122.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 269 kB 00:00 2024-09-13T11:51:18.740 INFO:teuthology.orchestra.run.smithi122.stdout:(4/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.4 MB/s | 863 kB 00:00 2024-09-13T11:51:18.869 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:08 ago on Fri 13 Sep 2024 11:50:10 AM UTC. 2024-09-13T11:51:18.890 INFO:teuthology.orchestra.run.smithi122.stdout:(5/12): ceph-mgr-dashboard-19.3.0-4697.gd58f730 21 MB/s | 14 MB 00:00 2024-09-13T11:51:18.924 INFO:teuthology.orchestra.run.smithi122.stdout:(6/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.8 MB/s | 1.0 MB 00:00 2024-09-13T11:51:18.991 INFO:teuthology.orchestra.run.smithi122.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 198 kB/s | 19 kB 00:00 2024-09-13T11:51:19.011 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:51:19.011 INFO:teuthology.orchestra.run.smithi100.stdout:======================================================================================= 2024-09-13T11:51:19.011 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-13T11:51:19.011 INFO:teuthology.orchestra.run.smithi100.stdout:======================================================================================= 2024-09-13T11:51:19.011 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T11:51:19.012 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 7.4 M 2024-09-13T11:51:19.012 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-13T11:51:19.012 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T11:51:19.012 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T11:51:19.012 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T11:51:19.012 INFO:teuthology.orchestra.run.smithi100.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T11:51:19.013 INFO:teuthology.orchestra.run.smithi100.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T11:51:19.013 INFO:teuthology.orchestra.run.smithi100.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T11:51:19.013 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T11:51:19.013 INFO:teuthology.orchestra.run.smithi100.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T11:51:19.013 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T11:51:19.013 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T11:51:19.014 INFO:teuthology.orchestra.run.smithi100.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T11:51:19.014 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:19.014 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T11:51:19.014 INFO:teuthology.orchestra.run.smithi100.stdout:======================================================================================= 2024-09-13T11:51:19.014 INFO:teuthology.orchestra.run.smithi100.stdout:Install 12 Packages 2024-09-13T11:51:19.014 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:19.015 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 38 M 2024-09-13T11:51:19.015 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 212 M 2024-09-13T11:51:19.015 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T11:51:19.024 INFO:teuthology.orchestra.run.smithi122.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.9 MB/s | 548 kB 00:00 2024-09-13T11:51:19.050 INFO:teuthology.orchestra.run.smithi122.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-13T11:51:19.075 INFO:teuthology.orchestra.run.smithi122.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.7 MB/s | 144 kB 00:00 2024-09-13T11:51:19.101 INFO:teuthology.orchestra.run.smithi122.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-13T11:51:19.159 INFO:teuthology.orchestra.run.smithi122.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 9.0 MB/s | 2.0 MB 00:00 2024-09-13T11:51:19.164 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:51:19.164 INFO:teuthology.orchestra.run.smithi122.stdout:Total 13 MB/s | 19 MB 00:01 2024-09-13T11:51:19.214 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T11:51:19.234 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T11:51:19.234 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T11:51:19.257 INFO:teuthology.orchestra.run.smithi100.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-13T11:51:19.323 INFO:teuthology.orchestra.run.smithi100.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.2 MB/s | 801 kB 00:00 2024-09-13T11:51:19.349 INFO:teuthology.orchestra.run.smithi100.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 356 kB/s | 33 kB 00:00 2024-09-13T11:51:19.391 INFO:teuthology.orchestra.run.smithi100.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 437 kB/s | 18 kB 00:00 2024-09-13T11:51:19.441 INFO:teuthology.orchestra.run.smithi100.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 774 kB/s | 38 kB 00:00 2024-09-13T11:51:19.510 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T11:51:19.511 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T11:51:19.583 INFO:teuthology.orchestra.run.smithi100.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-09-13T11:51:19.625 INFO:teuthology.orchestra.run.smithi100.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 5.7 MB/s | 245 kB 00:00 2024-09-13T11:51:19.700 INFO:teuthology.orchestra.run.smithi100.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 19 MB/s | 4.9 MB 00:00 2024-09-13T11:51:19.742 INFO:teuthology.orchestra.run.smithi100.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.9 MB/s | 250 kB 00:00 2024-09-13T11:51:19.817 INFO:teuthology.orchestra.run.smithi100.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 5.1 MB 00:00 2024-09-13T11:51:19.926 INFO:teuthology.orchestra.run.smithi100.stdout:(11/12): ceph-mgr-diskprediction-local-19.3.0-4 8.7 MB/s | 7.4 MB 00:00 2024-09-13T11:51:20.097 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T11:51:20.159 INFO:teuthology.orchestra.run.smithi100.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-09-13T11:51:20.160 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:51:20.160 INFO:teuthology.orchestra.run.smithi100.stdout:Total 33 MB/s | 38 MB 00:01 2024-09-13T11:51:20.176 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-13T11:51:20.211 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-13T11:51:20.258 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-13T11:51:20.298 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-13T11:51:20.399 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-13T11:51:20.515 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T11:51:20.531 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-13T11:51:20.541 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T11:51:20.541 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T11:51:20.633 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-13T11:51:20.677 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-13T11:51:20.884 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-13T11:51:20.954 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 10/12 2024-09-13T11:51:20.972 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T11:51:20.972 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T11:51:21.752 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T11:51:21.822 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T11:51:21.874 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T11:51:21.979 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T11:51:22.018 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T11:51:22.192 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T11:51:22.216 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T11:51:22.297 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T11:51:22.962 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T11:51:23.072 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T11:51:24.106 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 11/12 2024-09-13T11:51:24.175 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T11:51:24.317 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T11:51:25.353 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T11:51:25.382 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T11:51:25.989 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T11:51:25.989 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 1/12 2024-09-13T11:51:25.989 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 2/12 2024-09-13T11:51:25.989 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 3/12 2024-09-13T11:51:25.990 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-13T11:51:25.990 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-13T11:51:25.990 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-13T11:51:25.990 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-13T11:51:25.990 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-13T11:51:25.990 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-13T11:51:25.990 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-13T11:51:25.990 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-13T11:51:26.303 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-13T11:51:26.303 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:26.303 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T11:51:26.303 INFO:teuthology.orchestra.run.smithi122.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-13T11:51:26.303 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:51:26.303 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:51:26.303 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:51:26.304 INFO:teuthology.orchestra.run.smithi122.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-13T11:51:26.304 INFO:teuthology.orchestra.run.smithi122.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-13T11:51:26.304 INFO:teuthology.orchestra.run.smithi122.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-13T11:51:26.304 INFO:teuthology.orchestra.run.smithi122.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-13T11:51:26.304 INFO:teuthology.orchestra.run.smithi122.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-13T11:51:26.304 INFO:teuthology.orchestra.run.smithi122.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-13T11:51:26.304 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T11:51:26.304 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T11:51:26.304 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:26.304 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:51:26.589 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T11:51:27.147 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:16 ago on Fri 13 Sep 2024 11:50:11 AM UTC. 2024-09-13T11:51:27.279 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:51:27.281 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2024-09-13T11:51:27.281 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T11:51:27.281 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2024-09-13T11:51:27.281 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T11:51:27.281 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 7.4 M 2024-09-13T11:51:27.282 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T11:51:27.282 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T11:51:27.282 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T11:51:27.282 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T11:51:27.282 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T11:51:27.282 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T11:51:27.283 INFO:teuthology.orchestra.run.smithi122.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T11:51:27.283 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T11:51:27.283 INFO:teuthology.orchestra.run.smithi122.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T11:51:27.283 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T11:51:27.283 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T11:51:27.283 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T11:51:27.283 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:27.283 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T11:51:27.283 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2024-09-13T11:51:27.283 INFO:teuthology.orchestra.run.smithi122.stdout:Install 12 Packages 2024-09-13T11:51:27.283 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:27.284 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 38 M 2024-09-13T11:51:27.284 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 212 M 2024-09-13T11:51:27.284 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T11:51:27.897 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T11:51:27.897 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 1/12 2024-09-13T11:51:27.898 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T11:51:27.898 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T11:51:27.898 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T11:51:27.898 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T11:51:27.898 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T11:51:27.898 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T11:51:27.898 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T11:51:27.898 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T11:51:27.898 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T11:51:27.898 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T11:51:27.992 INFO:teuthology.orchestra.run.smithi122.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.2 MB/s | 191 kB 00:00 2024-09-13T11:51:28.026 INFO:teuthology.orchestra.run.smithi122.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.1 MB/s | 801 kB 00:00 2024-09-13T11:51:28.084 INFO:teuthology.orchestra.run.smithi122.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 357 kB/s | 33 kB 00:00 2024-09-13T11:51:28.168 INFO:teuthology.orchestra.run.smithi122.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 21 MB/s | 3.0 MB 00:00 2024-09-13T11:51:28.193 INFO:teuthology.orchestra.run.smithi122.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 167 kB/s | 18 kB 00:00 2024-09-13T11:51:28.244 INFO:teuthology.orchestra.run.smithi122.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 511 kB/s | 38 kB 00:00 2024-09-13T11:51:28.357 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T11:51:28.358 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:28.358 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T11:51:28.358 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:51:28.358 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T11:51:28.358 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T11:51:28.358 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T11:51:28.358 INFO:teuthology.orchestra.run.smithi100.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T11:51:28.358 INFO:teuthology.orchestra.run.smithi100.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T11:51:28.358 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T11:51:28.358 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T11:51:28.358 INFO:teuthology.orchestra.run.smithi100.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T11:51:28.358 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T11:51:28.359 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T11:51:28.359 INFO:teuthology.orchestra.run.smithi100.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T11:51:28.359 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:28.359 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:51:28.394 INFO:teuthology.orchestra.run.smithi122.stdout:(7/12): ceph-mgr-diskprediction-local-19.3.0-46 13 MB/s | 7.4 MB 00:00 2024-09-13T11:51:28.419 INFO:teuthology.orchestra.run.smithi122.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-09-13T11:51:28.494 INFO:teuthology.orchestra.run.smithi122.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 16 MB/s | 4.9 MB 00:00 2024-09-13T11:51:28.536 INFO:teuthology.orchestra.run.smithi122.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.1 MB/s | 250 kB 00:00 2024-09-13T11:51:28.687 INFO:teuthology.orchestra.run.smithi122.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 5.1 MB 00:00 2024-09-13T11:51:28.718 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-rook 2024-09-13T11:51:28.929 INFO:teuthology.orchestra.run.smithi122.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-09-13T11:51:28.929 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:51:28.930 INFO:teuthology.orchestra.run.smithi122.stdout:Total 23 MB/s | 38 MB 00:01 2024-09-13T11:51:29.278 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T11:51:29.287 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:19 ago on Fri 13 Sep 2024 11:50:10 AM UTC. 2024-09-13T11:51:29.303 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T11:51:29.303 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T11:51:29.420 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:51:29.421 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:29.421 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-13T11:51:29.421 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:29.421 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T11:51:29.421 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-rook noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 49 k 2024-09-13T11:51:29.421 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-13T11:51:29.422 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T11:51:29.422 INFO:teuthology.orchestra.run.smithi100.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T11:51:29.422 INFO:teuthology.orchestra.run.smithi100.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T11:51:29.422 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T11:51:29.422 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T11:51:29.422 INFO:teuthology.orchestra.run.smithi100.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T11:51:29.422 INFO:teuthology.orchestra.run.smithi100.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T11:51:29.422 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T11:51:29.422 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T11:51:29.422 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T11:51:29.422 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T11:51:29.422 INFO:teuthology.orchestra.run.smithi100.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T11:51:29.422 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:29.422 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T11:51:29.423 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:29.423 INFO:teuthology.orchestra.run.smithi100.stdout:Install 13 Packages 2024-09-13T11:51:29.423 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:29.423 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 2.2 M 2024-09-13T11:51:29.423 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 26 M 2024-09-13T11:51:29.423 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T11:51:29.666 INFO:teuthology.orchestra.run.smithi100.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 216 kB/s | 19 kB 00:00 2024-09-13T11:51:29.692 INFO:teuthology.orchestra.run.smithi100.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 224 kB/s | 26 kB 00:00 2024-09-13T11:51:29.740 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T11:51:29.740 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T11:51:29.750 INFO:teuthology.orchestra.run.smithi100.stdout:(3/13): ceph-mgr-rook-19.3.0-4697.gd58f7306.el9 285 kB/s | 49 kB 00:00 2024-09-13T11:51:29.776 INFO:teuthology.orchestra.run.smithi100.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-13T11:51:29.801 INFO:teuthology.orchestra.run.smithi100.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-13T11:51:29.826 INFO:teuthology.orchestra.run.smithi100.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-13T11:51:29.869 INFO:teuthology.orchestra.run.smithi100.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-13T11:51:29.902 INFO:teuthology.orchestra.run.smithi100.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 187 kB/s | 14 kB 00:00 2024-09-13T11:51:29.927 INFO:teuthology.orchestra.run.smithi100.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.5 MB/s | 279 kB 00:00 2024-09-13T11:51:29.961 INFO:teuthology.orchestra.run.smithi100.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.4 MB/s | 220 kB 00:00 2024-09-13T11:51:29.986 INFO:teuthology.orchestra.run.smithi100.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-13T11:51:30.011 INFO:teuthology.orchestra.run.smithi100.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 707 kB/s | 59 kB 00:00 2024-09-13T11:51:30.045 INFO:teuthology.orchestra.run.smithi100.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-13T11:51:30.047 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:51:30.047 INFO:teuthology.orchestra.run.smithi100.stdout:Total 3.5 MB/s | 2.2 MB 00:00 2024-09-13T11:51:30.110 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T11:51:30.125 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T11:51:30.125 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T11:51:30.213 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T11:51:30.214 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T11:51:30.543 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T11:51:30.622 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T11:51:30.660 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T11:51:30.750 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T11:51:30.758 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T11:51:30.803 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T11:51:30.924 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T11:51:30.979 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T11:51:30.986 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T11:51:31.010 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T11:51:31.034 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T11:51:31.072 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T11:51:31.090 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T11:51:31.113 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T11:51:31.215 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T11:51:31.325 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T11:51:31.381 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T11:51:31.753 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T11:51:31.881 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T11:51:32.092 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T11:51:32.145 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T11:51:32.179 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T11:51:32.243 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T11:51:32.266 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T11:51:33.036 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T11:51:33.036 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 1/13 2024-09-13T11:51:33.036 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T11:51:33.036 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T11:51:33.036 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T11:51:33.036 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T11:51:33.036 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T11:51:33.036 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T11:51:33.036 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T11:51:33.036 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T11:51:33.036 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T11:51:33.037 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T11:51:33.037 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T11:51:33.124 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T11:51:33.396 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T11:51:33.396 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:33.396 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T11:51:33.396 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:51:33.396 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T11:51:33.396 INFO:teuthology.orchestra.run.smithi100.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T11:51:33.396 INFO:teuthology.orchestra.run.smithi100.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T11:51:33.396 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T11:51:33.397 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T11:51:33.397 INFO:teuthology.orchestra.run.smithi100.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T11:51:33.397 INFO:teuthology.orchestra.run.smithi100.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T11:51:33.397 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T11:51:33.397 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T11:51:33.397 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T11:51:33.397 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T11:51:33.397 INFO:teuthology.orchestra.run.smithi100.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T11:51:33.397 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:33.397 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:51:33.645 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T11:51:34.158 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T11:51:34.185 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T11:51:34.206 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:24 ago on Fri 13 Sep 2024 11:50:10 AM UTC. 2024-09-13T11:51:34.339 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:51:34.339 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:34.339 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-13T11:51:34.340 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:34.340 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T11:51:34.340 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 154 k 2024-09-13T11:51:34.340 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-13T11:51:34.340 INFO:teuthology.orchestra.run.smithi100.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T11:51:34.340 INFO:teuthology.orchestra.run.smithi100.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T11:51:34.340 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T11:51:34.340 INFO:teuthology.orchestra.run.smithi100.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T11:51:34.340 INFO:teuthology.orchestra.run.smithi100.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T11:51:34.340 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:34.341 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T11:51:34.341 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:34.341 INFO:teuthology.orchestra.run.smithi100.stdout:Install 6 Packages 2024-09-13T11:51:34.341 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:34.342 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 7.0 M 2024-09-13T11:51:34.342 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 33 M 2024-09-13T11:51:34.342 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T11:51:34.605 INFO:teuthology.orchestra.run.smithi100.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 2.0 MB/s | 248 kB 00:00 2024-09-13T11:51:34.663 INFO:teuthology.orchestra.run.smithi100.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-13T11:51:34.689 INFO:teuthology.orchestra.run.smithi100.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.2 MB/s | 58 kB 00:00 2024-09-13T11:51:34.714 INFO:teuthology.orchestra.run.smithi100.stdout:(4/6): ceph-mgr-cephadm-19.3.0-4697.gd58f7306.e 667 kB/s | 154 kB 00:00 2024-09-13T11:51:34.739 INFO:teuthology.orchestra.run.smithi100.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-13T11:51:34.839 INFO:teuthology.orchestra.run.smithi100.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-09-13T11:51:34.841 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:51:34.841 INFO:teuthology.orchestra.run.smithi100.stdout:Total 14 MB/s | 7.0 MB 00:00 2024-09-13T11:51:34.911 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T11:51:34.922 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T11:51:34.922 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T11:51:35.021 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T11:51:35.021 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T11:51:35.333 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T11:51:35.488 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T11:51:35.532 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T11:51:36.052 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T11:51:36.143 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T11:51:36.211 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T11:51:36.237 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T11:51:36.817 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T11:51:36.817 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 1/12 2024-09-13T11:51:36.818 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T11:51:36.818 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T11:51:36.818 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T11:51:36.818 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T11:51:36.818 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T11:51:36.818 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T11:51:36.818 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T11:51:36.818 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T11:51:36.818 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T11:51:36.818 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T11:51:37.020 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T11:51:37.021 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/6 2024-09-13T11:51:37.021 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T11:51:37.021 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T11:51:37.021 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T11:51:37.021 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T11:51:37.260 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T11:51:37.260 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:37.260 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T11:51:37.260 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:51:37.260 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T11:51:37.260 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T11:51:37.260 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T11:51:37.260 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T11:51:37.261 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T11:51:37.261 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T11:51:37.261 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T11:51:37.261 INFO:teuthology.orchestra.run.smithi122.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T11:51:37.261 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T11:51:37.261 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T11:51:37.261 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T11:51:37.261 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:37.261 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:51:37.283 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T11:51:37.283 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:37.283 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T11:51:37.283 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:51:37.283 INFO:teuthology.orchestra.run.smithi100.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T11:51:37.283 INFO:teuthology.orchestra.run.smithi100.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T11:51:37.283 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T11:51:37.284 INFO:teuthology.orchestra.run.smithi100.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T11:51:37.284 INFO:teuthology.orchestra.run.smithi100.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T11:51:37.284 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:37.284 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:51:37.544 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-fuse 2024-09-13T11:51:37.669 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-rook 2024-09-13T11:51:38.109 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:28 ago on Fri 13 Sep 2024 11:50:10 AM UTC. 2024-09-13T11:51:38.241 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:51:38.242 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:38.242 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-13T11:51:38.242 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:38.243 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T11:51:38.243 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 900 k 2024-09-13T11:51:38.243 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-13T11:51:38.243 INFO:teuthology.orchestra.run.smithi100.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T11:51:38.243 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:38.243 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T11:51:38.243 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:38.243 INFO:teuthology.orchestra.run.smithi100.stdout:Install 2 Packages 2024-09-13T11:51:38.243 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:38.243 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 979 k 2024-09-13T11:51:38.243 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 2.9 M 2024-09-13T11:51:38.243 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T11:51:38.244 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:27 ago on Fri 13 Sep 2024 11:50:11 AM UTC. 2024-09-13T11:51:38.379 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:51:38.379 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:38.379 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T11:51:38.379 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:38.379 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T11:51:38.379 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 49 k 2024-09-13T11:51:38.379 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T11:51:38.380 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T11:51:38.380 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T11:51:38.380 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T11:51:38.380 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T11:51:38.380 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T11:51:38.380 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T11:51:38.380 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T11:51:38.380 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T11:51:38.380 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T11:51:38.380 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T11:51:38.380 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T11:51:38.380 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T11:51:38.381 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:38.381 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T11:51:38.381 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:38.381 INFO:teuthology.orchestra.run.smithi122.stdout:Install 13 Packages 2024-09-13T11:51:38.381 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:38.381 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 2.2 M 2024-09-13T11:51:38.381 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 26 M 2024-09-13T11:51:38.382 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T11:51:38.508 INFO:teuthology.orchestra.run.smithi100.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 535 kB/s | 80 kB 00:00 2024-09-13T11:51:38.610 INFO:teuthology.orchestra.run.smithi122.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 334 kB/s | 26 kB 00:00 2024-09-13T11:51:38.635 INFO:teuthology.orchestra.run.smithi122.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 188 kB/s | 19 kB 00:00 2024-09-13T11:51:38.658 INFO:teuthology.orchestra.run.smithi100.stdout:(2/2): ceph-fuse-19.3.0-4697.gd58f7306.el9.x86_ 2.9 MB/s | 900 kB 00:00 2024-09-13T11:51:38.658 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:51:38.659 INFO:teuthology.orchestra.run.smithi100.stdout:Total 2.3 MB/s | 979 kB 00:00 2024-09-13T11:51:38.671 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T11:51:38.693 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T11:51:38.693 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T11:51:38.710 INFO:teuthology.orchestra.run.smithi122.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-09-13T11:51:38.735 INFO:teuthology.orchestra.run.smithi122.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 222 kB 00:00 2024-09-13T11:51:38.761 INFO:teuthology.orchestra.run.smithi122.stdout:(5/13): ceph-mgr-rook-19.3.0-4697.gd58f7306.el9 217 kB/s | 49 kB 00:00 2024-09-13T11:51:38.773 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T11:51:38.774 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T11:51:38.786 INFO:teuthology.orchestra.run.smithi122.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-13T11:51:38.820 INFO:teuthology.orchestra.run.smithi122.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.5 MB/s | 279 kB 00:00 2024-09-13T11:51:38.870 INFO:teuthology.orchestra.run.smithi122.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 295 kB/s | 32 kB 00:00 2024-09-13T11:51:38.895 INFO:teuthology.orchestra.run.smithi122.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 129 kB/s | 14 kB 00:00 2024-09-13T11:51:38.920 INFO:teuthology.orchestra.run.smithi122.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-13T11:51:38.946 INFO:teuthology.orchestra.run.smithi122.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.9 MB/s | 220 kB 00:00 2024-09-13T11:51:38.961 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T11:51:38.971 INFO:teuthology.orchestra.run.smithi122.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-13T11:51:39.005 INFO:teuthology.orchestra.run.smithi122.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.7 MB/s | 1.0 MB 00:00 2024-09-13T11:51:39.007 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:51:39.007 INFO:teuthology.orchestra.run.smithi122.stdout:Total 3.5 MB/s | 2.2 MB 00:00 2024-09-13T11:51:39.071 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T11:51:39.085 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T11:51:39.085 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T11:51:39.085 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T11:51:39.157 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T11:51:39.175 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T11:51:39.176 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T11:51:39.673 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T11:51:39.826 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T11:51:39.826 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/2 2024-09-13T11:51:39.833 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T11:51:39.897 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T11:51:39.952 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T11:51:39.981 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T11:51:40.023 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T11:51:40.072 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T11:51:40.073 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:40.073 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T11:51:40.073 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T11:51:40.073 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:40.073 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:51:40.124 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T11:51:40.226 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T11:51:40.283 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T11:51:40.333 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-volume 2024-09-13T11:51:40.904 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:30 ago on Fri 13 Sep 2024 11:50:10 AM UTC. 2024-09-13T11:51:41.006 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T11:51:41.042 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:51:41.043 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:41.043 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-13T11:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T11:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-volume noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 285 k 2024-09-13T11:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T11:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-13T11:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 285 k 2024-09-13T11:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 1.3 M 2024-09-13T11:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T11:51:41.056 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T11:51:41.089 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T11:51:41.137 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T11:51:41.159 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T11:51:41.288 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-volume-19.3.0-4697.gd58f7306.el9.noarch.rp 1.1 MB/s | 285 kB 00:00 2024-09-13T11:51:41.289 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:51:41.289 INFO:teuthology.orchestra.run.smithi100.stdout:Total 1.1 MB/s | 285 kB 00:00 2024-09-13T11:51:41.289 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T11:51:41.297 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T11:51:41.297 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T11:51:41.329 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T11:51:41.330 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T11:51:41.544 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T11:51:41.615 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:51:41.631 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:51:41.631 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:51:41.631 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T11:51:41.631 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:41.924 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T11:51:41.925 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 1/13 2024-09-13T11:51:41.925 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T11:51:41.925 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T11:51:41.926 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T11:51:41.926 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T11:51:41.926 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T11:51:41.926 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T11:51:41.926 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T11:51:41.926 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T11:51:41.926 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T11:51:41.926 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T11:51:41.927 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T11:51:42.275 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T11:51:42.275 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:42.275 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T11:51:42.275 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:51:42.275 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T11:51:42.275 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T11:51:42.275 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T11:51:42.275 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T11:51:42.275 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T11:51:42.276 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T11:51:42.276 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T11:51:42.276 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T11:51:42.276 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T11:51:42.276 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T11:51:42.276 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T11:51:42.276 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T11:51:42.276 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:42.276 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:51:42.527 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T11:51:42.706 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:51:42.707 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:42.707 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T11:51:42.707 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:51:42.707 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:42.707 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:51:43.015 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install librados-devel 2024-09-13T11:51:43.090 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:32 ago on Fri 13 Sep 2024 11:50:11 AM UTC. 2024-09-13T11:51:43.224 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:51:43.225 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:43.225 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T11:51:43.225 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:43.225 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T11:51:43.225 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 154 k 2024-09-13T11:51:43.225 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T11:51:43.226 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T11:51:43.226 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T11:51:43.226 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T11:51:43.226 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T11:51:43.226 INFO:teuthology.orchestra.run.smithi122.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T11:51:43.226 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:43.226 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T11:51:43.226 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:43.226 INFO:teuthology.orchestra.run.smithi122.stdout:Install 6 Packages 2024-09-13T11:51:43.226 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:43.227 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 7.0 M 2024-09-13T11:51:43.227 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 33 M 2024-09-13T11:51:43.227 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T11:51:43.585 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:33 ago on Fri 13 Sep 2024 11:50:10 AM UTC. 2024-09-13T11:51:43.599 INFO:teuthology.orchestra.run.smithi122.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-09-13T11:51:43.623 INFO:teuthology.orchestra.run.smithi122.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4697.gd58f7306.e 741 kB/s | 154 kB 00:00 2024-09-13T11:51:43.674 INFO:teuthology.orchestra.run.smithi122.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-09-13T11:51:43.699 INFO:teuthology.orchestra.run.smithi122.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 765 kB/s | 58 kB 00:00 2024-09-13T11:51:43.721 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:51:43.722 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:43.722 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-09-13T11:51:43.722 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:43.722 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T11:51:43.722 INFO:teuthology.orchestra.run.smithi100.stdout: librados-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 127 k 2024-09-13T11:51:43.722 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:43.722 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T11:51:43.723 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:43.723 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-13T11:51:43.723 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:43.723 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 127 k 2024-09-13T11:51:43.723 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 456 k 2024-09-13T11:51:43.723 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T11:51:43.724 INFO:teuthology.orchestra.run.smithi122.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-13T11:51:43.936 INFO:teuthology.orchestra.run.smithi100.stdout:librados-devel-19.3.0-4697.gd58f7306.el9.x86_64 597 kB/s | 127 kB 00:00 2024-09-13T11:51:43.936 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:51:43.936 INFO:teuthology.orchestra.run.smithi100.stdout:Total 592 kB/s | 127 kB 00:00 2024-09-13T11:51:43.937 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T11:51:43.942 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T11:51:43.942 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T11:51:44.006 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T11:51:44.006 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T11:51:44.157 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T11:51:44.266 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:51:44.676 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:51:44.783 INFO:teuthology.orchestra.run.smithi122.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.4 MB/s | 6.0 MB 00:01 2024-09-13T11:51:44.785 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:51:44.786 INFO:teuthology.orchestra.run.smithi122.stdout:Total 4.5 MB/s | 7.0 MB 00:01 2024-09-13T11:51:44.857 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T11:51:44.868 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T11:51:44.869 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T11:51:44.927 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:51:44.927 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:44.927 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T11:51:44.927 INFO:teuthology.orchestra.run.smithi100.stdout: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:51:44.927 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:44.927 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:51:44.969 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T11:51:44.970 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T11:51:45.169 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install libcephfs2 2024-09-13T11:51:45.275 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T11:51:45.449 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T11:51:45.501 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T11:51:45.746 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:35 ago on Fri 13 Sep 2024 11:50:10 AM UTC. 2024-09-13T11:51:45.837 INFO:teuthology.orchestra.run.smithi100.stdout:Package libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:51:45.882 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:51:45.883 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-13T11:51:45.883 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:51:45.975 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install libcephfs-devel 2024-09-13T11:51:46.025 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T11:51:46.112 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T11:51:46.181 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T11:51:46.205 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T11:51:46.536 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:36 ago on Fri 13 Sep 2024 11:50:10 AM UTC. 2024-09-13T11:51:46.669 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:51:46.670 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:46.670 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-09-13T11:51:46.670 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:46.670 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T11:51:46.670 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 32 k 2024-09-13T11:51:46.670 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:46.670 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T11:51:46.670 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:46.670 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-13T11:51:46.671 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:46.671 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 32 k 2024-09-13T11:51:46.671 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 142 k 2024-09-13T11:51:46.671 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T11:51:46.814 INFO:teuthology.orchestra.run.smithi100.stdout:libcephfs-devel-19.3.0-4697.gd58f7306.el9.x86_6 220 kB/s | 32 kB 00:00 2024-09-13T11:51:46.815 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:51:46.815 INFO:teuthology.orchestra.run.smithi100.stdout:Total 218 kB/s | 32 kB 00:00 2024-09-13T11:51:46.815 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T11:51:46.819 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T11:51:46.819 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T11:51:46.841 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T11:51:46.841 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T11:51:46.932 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T11:51:47.006 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T11:51:47.006 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/6 2024-09-13T11:51:47.006 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T11:51:47.007 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T11:51:47.007 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T11:51:47.007 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T11:51:47.044 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:51:47.273 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T11:51:47.274 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:47.274 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T11:51:47.274 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:51:47.274 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T11:51:47.274 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T11:51:47.274 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T11:51:47.274 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T11:51:47.274 INFO:teuthology.orchestra.run.smithi122.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T11:51:47.274 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:47.274 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:51:47.401 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:51:47.548 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-fuse 2024-09-13T11:51:47.625 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:51:47.626 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:47.626 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T11:51:47.626 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:51:47.626 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:47.626 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:51:47.818 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install librados2 2024-09-13T11:51:48.111 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:37 ago on Fri 13 Sep 2024 11:50:11 AM UTC. 2024-09-13T11:51:48.244 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:51:48.245 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:48.245 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T11:51:48.245 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:48.245 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T11:51:48.245 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 900 k 2024-09-13T11:51:48.245 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T11:51:48.245 INFO:teuthology.orchestra.run.smithi122.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T11:51:48.245 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:48.246 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T11:51:48.246 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:48.246 INFO:teuthology.orchestra.run.smithi122.stdout:Install 2 Packages 2024-09-13T11:51:48.246 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:48.246 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 979 k 2024-09-13T11:51:48.246 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 2.9 M 2024-09-13T11:51:48.246 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T11:51:48.382 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:38 ago on Fri 13 Sep 2024 11:50:10 AM UTC. 2024-09-13T11:51:48.473 INFO:teuthology.orchestra.run.smithi100.stdout:Package librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:51:48.516 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:51:48.517 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-13T11:51:48.517 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:51:48.582 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install librbd1 2024-09-13T11:51:48.610 INFO:teuthology.orchestra.run.smithi122.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 715 kB/s | 80 kB 00:00 2024-09-13T11:51:48.760 INFO:teuthology.orchestra.run.smithi122.stdout:(2/2): ceph-fuse-19.3.0-4697.gd58f7306.el9.x86_ 3.4 MB/s | 900 kB 00:00 2024-09-13T11:51:48.760 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:51:48.761 INFO:teuthology.orchestra.run.smithi122.stdout:Total 1.9 MB/s | 979 kB 00:00 2024-09-13T11:51:48.772 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T11:51:48.794 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T11:51:48.794 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T11:51:48.874 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T11:51:48.874 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T11:51:49.069 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T11:51:49.141 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:39 ago on Fri 13 Sep 2024 11:50:10 AM UTC. 2024-09-13T11:51:49.203 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T11:51:49.231 INFO:teuthology.orchestra.run.smithi100.stdout:Package librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:51:49.267 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T11:51:49.275 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:51:49.276 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-13T11:51:49.277 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:51:49.342 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-rados 2024-09-13T11:51:49.909 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:39 ago on Fri 13 Sep 2024 11:50:10 AM UTC. 2024-09-13T11:51:49.994 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:51:50.001 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T11:51:50.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/2 2024-09-13T11:51:50.037 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:51:50.038 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-13T11:51:50.038 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:51:50.107 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-rgw 2024-09-13T11:51:50.294 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T11:51:50.295 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:50.295 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T11:51:50.295 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T11:51:50.295 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:50.295 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:51:50.571 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-volume 2024-09-13T11:51:50.664 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:40 ago on Fri 13 Sep 2024 11:50:10 AM UTC. 2024-09-13T11:51:50.754 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:51:50.797 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:51:50.798 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-13T11:51:50.798 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:51:50.866 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-cephfs 2024-09-13T11:51:51.145 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:40 ago on Fri 13 Sep 2024 11:50:11 AM UTC. 2024-09-13T11:51:51.286 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:51:51.286 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:51.287 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T11:51:51.287 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:51.287 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T11:51:51.287 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-volume noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 285 k 2024-09-13T11:51:51.287 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:51.287 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T11:51:51.287 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:51.287 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-13T11:51:51.287 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:51.287 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 285 k 2024-09-13T11:51:51.287 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 1.3 M 2024-09-13T11:51:51.287 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T11:51:51.421 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:41 ago on Fri 13 Sep 2024 11:50:10 AM UTC. 2024-09-13T11:51:51.511 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:51:51.523 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-volume-19.3.0-4697.gd58f7306.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-13T11:51:51.523 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:51:51.524 INFO:teuthology.orchestra.run.smithi122.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-13T11:51:51.524 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T11:51:51.532 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T11:51:51.533 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T11:51:51.554 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:51:51.555 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-13T11:51:51.555 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:51:51.565 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T11:51:51.565 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T11:51:51.621 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-rbd 2024-09-13T11:51:51.787 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T11:51:51.850 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:51:51.867 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:51:51.867 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:51:51.867 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T11:51:51.867 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:52.185 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:42 ago on Fri 13 Sep 2024 11:50:10 AM UTC. 2024-09-13T11:51:52.274 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:51:52.318 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:51:52.319 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-13T11:51:52.319 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:51:52.382 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install rbd-fuse 2024-09-13T11:51:52.854 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:51:52.855 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:52.855 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T11:51:52.855 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:51:52.855 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:52.855 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:51:52.948 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:42 ago on Fri 13 Sep 2024 11:50:10 AM UTC. 2024-09-13T11:51:53.091 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:51:53.092 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:53.092 INFO:teuthology.orchestra.run.smithi100.stdout: Package Architecture Version Repository Size 2024-09-13T11:51:53.092 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:53.092 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T11:51:53.092 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 85 k 2024-09-13T11:51:53.092 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:53.093 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T11:51:53.093 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:53.093 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-13T11:51:53.093 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:53.093 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 85 k 2024-09-13T11:51:53.093 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 231 k 2024-09-13T11:51:53.093 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T11:51:53.153 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librados-devel 2024-09-13T11:51:53.276 INFO:teuthology.orchestra.run.smithi100.stdout:rbd-fuse-19.3.0-4697.gd58f7306.el9.x86_64.rpm 466 kB/s | 85 kB 00:00 2024-09-13T11:51:53.277 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:51:53.277 INFO:teuthology.orchestra.run.smithi100.stdout:Total 463 kB/s | 85 kB 00:00 2024-09-13T11:51:53.277 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T11:51:53.283 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T11:51:53.283 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T11:51:53.347 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T11:51:53.347 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T11:51:53.498 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T11:51:53.632 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:51:53.731 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:42 ago on Fri 13 Sep 2024 11:50:11 AM UTC. 2024-09-13T11:51:53.866 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:51:53.867 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:53.867 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-09-13T11:51:53.867 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:53.867 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T11:51:53.867 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 127 k 2024-09-13T11:51:53.867 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:53.867 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T11:51:53.867 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:53.868 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-13T11:51:53.868 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:53.868 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 127 k 2024-09-13T11:51:53.868 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 456 k 2024-09-13T11:51:53.868 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T11:51:53.991 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:51:54.078 INFO:teuthology.orchestra.run.smithi122.stdout:librados-devel-19.3.0-4697.gd58f7306.el9.x86_64 602 kB/s | 127 kB 00:00 2024-09-13T11:51:54.079 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:51:54.079 INFO:teuthology.orchestra.run.smithi122.stdout:Total 598 kB/s | 127 kB 00:00 2024-09-13T11:51:54.079 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T11:51:54.084 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T11:51:54.085 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T11:51:54.148 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T11:51:54.149 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T11:51:54.237 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:51:54.237 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:54.237 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T11:51:54.237 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:51:54.237 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:54.237 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:51:54.302 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T11:51:54.401 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:51:54.530 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install rbd-mirror 2024-09-13T11:51:54.879 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:51:55.103 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:45 ago on Fri 13 Sep 2024 11:50:10 AM UTC. 2024-09-13T11:51:55.140 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:51:55.140 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:55.140 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T11:51:55.140 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:51:55.140 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:55.140 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:51:55.239 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:51:55.239 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:55.239 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-09-13T11:51:55.240 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:55.240 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T11:51:55.240 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-mirror x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T11:51:55.240 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:55.240 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T11:51:55.240 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:55.240 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-13T11:51:55.240 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:55.241 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 3.2 M 2024-09-13T11:51:55.241 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 14 M 2024-09-13T11:51:55.241 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T11:51:55.397 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install libcephfs2 2024-09-13T11:51:55.588 INFO:teuthology.orchestra.run.smithi100.stdout:rbd-mirror-19.3.0-4697.gd58f7306.el9.x86_64.rpm 9.1 MB/s | 3.2 MB 00:00 2024-09-13T11:51:55.588 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:51:55.589 INFO:teuthology.orchestra.run.smithi100.stdout:Total 9.1 MB/s | 3.2 MB 00:00 2024-09-13T11:51:55.589 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T11:51:55.597 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T11:51:55.598 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T11:51:55.684 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T11:51:55.684 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T11:51:55.975 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:44 ago on Fri 13 Sep 2024 11:50:11 AM UTC. 2024-09-13T11:51:56.066 INFO:teuthology.orchestra.run.smithi122.stdout:Package libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:51:56.109 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:51:56.110 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T11:51:56.110 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:51:56.143 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T11:51:56.187 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install libcephfs-devel 2024-09-13T11:51:56.213 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:51:56.239 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:51:56.239 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:51:56.239 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T11:51:56.239 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T11:51:56.239 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T11:51:56.240 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:56.749 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:45 ago on Fri 13 Sep 2024 11:50:11 AM UTC. 2024-09-13T11:51:56.883 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:51:56.883 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:56.883 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-09-13T11:51:56.883 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:56.883 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T11:51:56.883 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 32 k 2024-09-13T11:51:56.883 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:56.883 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T11:51:56.884 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:51:56.884 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-13T11:51:56.884 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:56.884 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 32 k 2024-09-13T11:51:56.884 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 142 k 2024-09-13T11:51:56.884 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T11:51:57.032 INFO:teuthology.orchestra.run.smithi122.stdout:libcephfs-devel-19.3.0-4697.gd58f7306.el9.x86_6 213 kB/s | 32 kB 00:00 2024-09-13T11:51:57.033 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:51:57.033 INFO:teuthology.orchestra.run.smithi122.stdout:Total 211 kB/s | 32 kB 00:00 2024-09-13T11:51:57.033 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T11:51:57.037 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T11:51:57.037 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T11:51:57.054 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T11:51:57.054 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T11:51:57.141 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T11:51:57.263 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:51:57.387 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:51:57.387 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:57.387 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T11:51:57.388 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:51:57.388 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:57.388 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:51:57.625 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install rbd-nbd 2024-09-13T11:51:57.660 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:51:57.891 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:51:57.891 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:57.891 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T11:51:57.891 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:51:57.891 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:51:57.891 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:51:58.082 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librados2 2024-09-13T11:51:58.197 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:48 ago on Fri 13 Sep 2024 11:50:10 AM UTC. 2024-09-13T11:51:58.331 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T11:51:58.331 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:58.331 INFO:teuthology.orchestra.run.smithi100.stdout: Package Architecture Version Repository Size 2024-09-13T11:51:58.331 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:58.332 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T11:51:58.332 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-nbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 170 k 2024-09-13T11:51:58.332 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:58.332 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T11:51:58.332 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T11:51:58.332 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-13T11:51:58.332 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:58.332 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 170 k 2024-09-13T11:51:58.332 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 490 k 2024-09-13T11:51:58.332 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T11:51:58.533 INFO:teuthology.orchestra.run.smithi100.stdout:rbd-nbd-19.3.0-4697.gd58f7306.el9.x86_64.rpm 849 kB/s | 170 kB 00:00 2024-09-13T11:51:58.534 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:51:58.534 INFO:teuthology.orchestra.run.smithi100.stdout:Total 842 kB/s | 170 kB 00:00 2024-09-13T11:51:58.534 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T11:51:58.542 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T11:51:58.542 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T11:51:58.607 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T11:51:58.607 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T11:51:58.643 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:47 ago on Fri 13 Sep 2024 11:50:11 AM UTC. 2024-09-13T11:51:58.733 INFO:teuthology.orchestra.run.smithi122.stdout:Package librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:51:58.764 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T11:51:58.776 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:51:58.777 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T11:51:58.777 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:51:58.848 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librbd1 2024-09-13T11:51:58.873 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:51:59.291 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:51:59.409 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:48 ago on Fri 13 Sep 2024 11:50:11 AM UTC. 2024-09-13T11:51:59.499 INFO:teuthology.orchestra.run.smithi122.stdout:Package librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:51:59.539 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:51:59.539 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:59.539 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T11:51:59.539 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:51:59.539 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:51:59.539 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T11:51:59.543 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:51:59.544 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T11:51:59.544 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:51:59.614 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rados 2024-09-13T11:51:59.781 DEBUG:teuthology.parallel:result is None 2024-09-13T11:52:00.219 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:49 ago on Fri 13 Sep 2024 11:50:11 AM UTC. 2024-09-13T11:52:00.308 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:52:00.351 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:52:00.352 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T11:52:00.352 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:52:00.424 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rgw 2024-09-13T11:52:00.993 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:49 ago on Fri 13 Sep 2024 11:50:11 AM UTC. 2024-09-13T11:52:01.082 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:52:01.126 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:52:01.127 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T11:52:01.127 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:52:01.201 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-cephfs 2024-09-13T11:52:01.759 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:50 ago on Fri 13 Sep 2024 11:50:11 AM UTC. 2024-09-13T11:52:01.849 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:52:01.892 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:52:01.893 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T11:52:01.893 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:52:01.964 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rbd 2024-09-13T11:52:02.527 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:51 ago on Fri 13 Sep 2024 11:50:11 AM UTC. 2024-09-13T11:52:02.616 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:52:02.660 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:52:02.661 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T11:52:02.661 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:52:02.732 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-fuse 2024-09-13T11:52:03.291 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:52 ago on Fri 13 Sep 2024 11:50:11 AM UTC. 2024-09-13T11:52:03.425 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:52:03.426 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:52:03.426 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-09-13T11:52:03.426 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:52:03.426 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T11:52:03.426 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 85 k 2024-09-13T11:52:03.426 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:03.426 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T11:52:03.426 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:52:03.426 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-13T11:52:03.426 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:03.426 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 85 k 2024-09-13T11:52:03.426 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 231 k 2024-09-13T11:52:03.427 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T11:52:03.619 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-fuse-19.3.0-4697.gd58f7306.el9.x86_64.rpm 443 kB/s | 85 kB 00:00 2024-09-13T11:52:03.619 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:52:03.620 INFO:teuthology.orchestra.run.smithi122.stdout:Total 440 kB/s | 85 kB 00:00 2024-09-13T11:52:03.620 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T11:52:03.630 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T11:52:03.630 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T11:52:03.689 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T11:52:03.690 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T11:52:03.837 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T11:52:03.949 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:52:04.316 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:52:04.564 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:52:04.564 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:04.564 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T11:52:04.564 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:52:04.564 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:04.564 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:52:04.848 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-mirror 2024-09-13T11:52:05.433 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:54 ago on Fri 13 Sep 2024 11:50:11 AM UTC. 2024-09-13T11:52:05.568 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:52:05.569 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:52:05.569 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-09-13T11:52:05.569 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:52:05.569 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T11:52:05.570 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T11:52:05.570 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:05.570 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T11:52:05.570 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:52:05.570 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-13T11:52:05.570 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:05.570 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 3.2 M 2024-09-13T11:52:05.570 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 14 M 2024-09-13T11:52:05.570 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T11:52:05.914 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-mirror-19.3.0-4697.gd58f7306.el9.x86_64.rpm 9.2 MB/s | 3.2 MB 00:00 2024-09-13T11:52:05.915 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:52:05.915 INFO:teuthology.orchestra.run.smithi122.stdout:Total 9.2 MB/s | 3.2 MB 00:00 2024-09-13T11:52:05.916 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T11:52:05.924 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T11:52:05.924 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T11:52:06.010 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T11:52:06.011 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T11:52:06.454 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T11:52:06.507 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:52:06.532 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:52:06.532 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:52:06.532 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T11:52:06.533 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T11:52:06.533 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T11:52:06.533 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:07.631 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:52:07.632 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:07.632 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T11:52:07.632 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:52:07.632 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:07.632 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:52:07.895 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-nbd 2024-09-13T11:52:08.477 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:57 ago on Fri 13 Sep 2024 11:50:11 AM UTC. 2024-09-13T11:52:08.613 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T11:52:08.614 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:52:08.614 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-09-13T11:52:08.614 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:52:08.614 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T11:52:08.614 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 170 k 2024-09-13T11:52:08.615 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:08.615 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T11:52:08.615 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T11:52:08.615 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-13T11:52:08.615 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:08.615 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 170 k 2024-09-13T11:52:08.615 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 490 k 2024-09-13T11:52:08.615 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T11:52:08.826 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-nbd-19.3.0-4697.gd58f7306.el9.x86_64.rpm 799 kB/s | 170 kB 00:00 2024-09-13T11:52:08.827 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:52:08.827 INFO:teuthology.orchestra.run.smithi122.stdout:Total 794 kB/s | 170 kB 00:00 2024-09-13T11:52:08.827 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T11:52:08.835 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T11:52:08.835 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T11:52:08.901 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T11:52:08.901 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T11:52:09.055 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T11:52:09.157 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:52:09.575 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:52:09.842 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:52:09.843 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:09.843 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T11:52:09.843 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:52:09.843 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:09.843 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T11:52:10.090 DEBUG:teuthology.parallel:result is None 2024-09-13T11:52:10.091 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:52:10.248 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T11:52:10.274 INFO:teuthology.orchestra.run.smithi100.stdout:19.3.0-4697.gd58f7306.el9 2024-09-13T11:52:10.275 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4697.gd58f7306.el9 2024-09-13T11:52:10.275 INFO:teuthology.task.install:The correct ceph version 19.3.0-4697.gd58f7306 is installed. 2024-09-13T11:52:10.277 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:52:10.427 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T11:52:10.449 INFO:teuthology.orchestra.run.smithi122.stdout:19.3.0-4697.gd58f7306.el9 2024-09-13T11:52:10.450 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4697.gd58f7306.el9 2024-09-13T11:52:10.450 INFO:teuthology.task.install:The correct ceph version 19.3.0-4697.gd58f7306 is installed. 2024-09-13T11:52:10.452 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-13T11:52:10.452 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T11:52:10.453 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T11:52:10.483 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T11:52:10.483 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T11:52:10.516 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-13T11:52:10.516 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T11:52:10.516 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T11:52:10.553 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T11:52:10.621 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T11:52:10.621 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T11:52:10.646 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T11:52:10.714 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-13T11:52:10.714 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T11:52:10.714 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T11:52:10.744 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T11:52:10.811 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T11:52:10.811 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T11:52:10.836 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T11:52:10.900 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-13T11:52:10.900 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T11:52:10.900 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T11:52:10.924 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T11:52:10.990 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T11:52:10.990 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T11:52:11.014 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T11:52:11.079 INFO:teuthology.run_tasks:Running task ceph... 2024-09-13T11:52:11.172 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-13T11:52:11.172 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 777 /var/log/ceph 2024-09-13T11:52:11.175 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 777 /var/log/ceph 2024-09-13T11:52:11.200 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-13T11:52:11.200 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T11:52:11.243 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T11:52:11.267 INFO:tasks.ceph:Creating extra log directories... 2024-09-13T11:52:11.267 DEBUG:teuthology.orchestra.run.smithi100:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T11:52:11.307 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T11:52:11.341 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-13T11:52:11.342 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'slow request', 'overall HEALTH_'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-13T11:52:11.342 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903450', 'branch': 'wip-aclamk-cbt-combined', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest} thrashers/force-sync-many workloads/pool-create-delete}', 'email': 'akupczyk@redhat.com', 'first_in_suite': False, 'job_id': '7903450', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-aclamk-cbt-combined'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'slow request', 'overall HEALTH_'], 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-aclamk-cbt-combined', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'owner': 'scheduled_akupczyk@teuthology', 'priority': 160, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 4159, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'sleep_before_teardown': 0, 'subset': '682/13280', 'suite': 'rados', 'suite_branch': 'wip-aclamk-cbt-combined', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'targets': {'smithi100.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA7IK4oIiP/yLkpdkFlGdYRURPZB2n6+YbXllR3K/X5MgwPuWVHGOmMltw310yG3Z5B5gmroJmL6iSPgbhjKXqo=', 'smithi122.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCpNMI+SeC5pSpCvsbC+7/y+mxh21V3UdMtQYMYajSz/SLni01bqC/nB7sO6MYmgUE7OCFXuz5CxQFAJMD1pXcs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'store_thrash': True, 'thrash_delay': 1, 'thrash_many': True}}, {'exec': {'client.0': ['ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-13_09:43:51', 'tube': 'smithi', 'user': 'akupczyk', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-13T11:52:11.342 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T11:52:11.373 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T11:52:11.387 DEBUG:teuthology.orchestra.run.smithi100:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T11:52:11.427 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T11:52:11.453 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T11:52:11.453 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T11:52:11.506 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T11:52:11.506 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_1 2024-09-13T11:52:11.561 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T11:52:11.561 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:52:11.561 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-13T11:52:11.561 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:52:11.562 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T11:52:11.562 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-09-13 11:51:56.508859213 +0000 2024-09-13T11:52:11.562 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-09-13 11:49:15.134750422 +0000 2024-09-13T11:52:11.562 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-09-13 11:49:15.134750422 +0000 2024-09-13T11:52:11.562 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-09-13 11:49:15.134750422 +0000 2024-09-13T11:52:11.562 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T11:52:11.624 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-09-13T11:52:11.624 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-09-13T11:52:11.624 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000184367 s, 2.8 MB/s 2024-09-13T11:52:11.625 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T11:52:11.681 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_2 2024-09-13T11:52:11.736 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T11:52:11.736 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:52:11.737 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-13T11:52:11.737 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:52:11.737 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T11:52:11.737 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-09-13 11:51:56.508859213 +0000 2024-09-13T11:52:11.737 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-09-13 11:49:15.450742150 +0000 2024-09-13T11:52:11.737 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-09-13 11:49:15.450742150 +0000 2024-09-13T11:52:11.737 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-09-13 11:49:15.450742150 +0000 2024-09-13T11:52:11.737 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T11:52:11.799 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-09-13T11:52:11.799 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-09-13T11:52:11.799 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000223556 s, 2.3 MB/s 2024-09-13T11:52:11.801 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T11:52:11.858 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_3 2024-09-13T11:52:11.912 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T11:52:11.912 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:52:11.912 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-13T11:52:11.912 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:52:11.912 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T11:52:11.913 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-09-13 11:51:56.508859213 +0000 2024-09-13T11:52:11.913 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-09-13 11:49:15.798733040 +0000 2024-09-13T11:52:11.913 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-09-13 11:49:15.798733040 +0000 2024-09-13T11:52:11.913 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-09-13 11:49:15.798733040 +0000 2024-09-13T11:52:11.913 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T11:52:11.975 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-09-13T11:52:11.976 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-09-13T11:52:11.976 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000221231 s, 2.3 MB/s 2024-09-13T11:52:11.977 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T11:52:12.033 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_4 2024-09-13T11:52:12.088 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T11:52:12.088 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:52:12.088 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-13T11:52:12.088 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:52:12.088 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T11:52:12.089 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-09-13 11:51:56.509859188 +0000 2024-09-13T11:52:12.089 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-09-13 11:49:16.120724611 +0000 2024-09-13T11:52:12.089 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-09-13 11:49:16.120724611 +0000 2024-09-13T11:52:12.089 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-09-13 11:49:16.120724611 +0000 2024-09-13T11:52:12.089 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T11:52:12.151 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-09-13T11:52:12.151 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-09-13T11:52:12.151 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000161785 s, 3.2 MB/s 2024-09-13T11:52:12.152 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T11:52:12.208 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T11:52:12.208 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T11:52:12.208 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T11:52:12.226 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T11:52:12.226 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_1 2024-09-13T11:52:12.282 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T11:52:12.282 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:52:12.283 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-13T11:52:12.283 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:52:12.283 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T11:52:12.283 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-09-13 11:52:06.789558292 +0000 2024-09-13T11:52:12.283 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-09-13 11:49:15.125172350 +0000 2024-09-13T11:52:12.283 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-09-13 11:49:15.125172350 +0000 2024-09-13T11:52:12.283 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-09-13 11:49:15.125172350 +0000 2024-09-13T11:52:12.283 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T11:52:12.347 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-09-13T11:52:12.347 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-09-13T11:52:12.348 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000222702 s, 2.3 MB/s 2024-09-13T11:52:12.349 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T11:52:12.406 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_2 2024-09-13T11:52:12.461 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T11:52:12.462 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:52:12.462 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-13T11:52:12.462 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:52:12.462 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T11:52:12.462 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-09-13 11:52:06.789558292 +0000 2024-09-13T11:52:12.462 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-09-13 11:49:15.464164646 +0000 2024-09-13T11:52:12.462 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-09-13 11:49:15.464164646 +0000 2024-09-13T11:52:12.462 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-09-13 11:49:15.464164646 +0000 2024-09-13T11:52:12.462 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T11:52:12.532 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-09-13T11:52:12.532 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-09-13T11:52:12.532 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000233389 s, 2.2 MB/s 2024-09-13T11:52:12.532 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T11:52:12.589 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_3 2024-09-13T11:52:12.647 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T11:52:12.647 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:52:12.647 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-13T11:52:12.647 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:52:12.648 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T11:52:12.648 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-09-13 11:52:06.790558269 +0000 2024-09-13T11:52:12.648 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-09-13 11:49:15.793157170 +0000 2024-09-13T11:52:12.648 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-09-13 11:49:15.793157170 +0000 2024-09-13T11:52:12.648 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-09-13 11:49:15.793157170 +0000 2024-09-13T11:52:12.648 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T11:52:12.712 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-09-13T11:52:12.712 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-09-13T11:52:12.712 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000168595 s, 3.0 MB/s 2024-09-13T11:52:12.713 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T11:52:12.769 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_4 2024-09-13T11:52:12.825 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T11:52:12.826 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:52:12.826 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-13T11:52:12.826 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:52:12.826 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T11:52:12.826 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-09-13 11:52:06.790558269 +0000 2024-09-13T11:52:12.826 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-09-13 11:49:16.130149512 +0000 2024-09-13T11:52:12.826 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-09-13 11:49:16.130149512 +0000 2024-09-13T11:52:12.826 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-09-13 11:49:16.130149512 +0000 2024-09-13T11:52:12.826 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T11:52:12.889 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-09-13T11:52:12.889 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-09-13T11:52:12.889 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000195989 s, 2.6 MB/s 2024-09-13T11:52:12.891 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T11:52:12.947 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-13T11:52:12.947 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi100.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi122.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-13T11:52:12.947 INFO:tasks.ceph:Generating config... 2024-09-13T11:52:12.948 INFO:tasks.ceph:[client] debug monc = 20 2024-09-13T11:52:12.948 INFO:tasks.ceph:[client] debug ms = 1 2024-09-13T11:52:12.948 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-13T11:52:12.948 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-13T11:52:12.949 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-13T11:52:12.949 INFO:tasks.ceph:[global] ms type = async 2024-09-13T11:52:12.949 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-13T11:52:12.949 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-13T11:52:12.949 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-13T11:52:12.949 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-13T11:52:12.949 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-13T11:52:12.949 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-13T11:52:12.949 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-13T11:52:12.949 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-13T11:52:12.949 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-13T11:52:12.949 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-13T11:52:12.949 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-13T11:52:12.949 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-13T11:52:12.950 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-13T11:52:12.950 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-13T11:52:12.950 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-13T11:52:12.950 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-13T11:52:12.950 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-13T11:52:12.950 INFO:tasks.ceph:[osd] bluestore write v2 random = True 2024-09-13T11:52:12.950 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-13T11:52:12.950 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-13T11:52:12.950 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-13T11:52:12.950 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-13T11:52:12.950 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-13T11:52:12.950 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-13T11:52:12.950 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-13T11:52:12.950 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-13T11:52:12.951 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-13T11:52:12.951 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-13T11:52:12.951 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-13T11:52:12.951 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-13T11:52:12.951 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-13T11:52:12.951 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-13T11:52:12.951 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-13T11:52:12.951 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-13T11:52:12.951 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-13T11:52:12.951 INFO:tasks.ceph:Setting up mon.a... 2024-09-13T11:52:12.951 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-13T11:52:13.029 INFO:teuthology.orchestra.run.smithi100.stdout:creating /etc/ceph/ceph.keyring 2024-09-13T11:52:13.032 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-13T11:52:13.077 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T11:52:13.143 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.100'), ('mon.c', '[v2:172.21.15.100:3301,v1:172.21.15.100:6790]'), ('mon.b', '172.21.15.122')] 2024-09-13T11:52:13.143 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.100,[v2:172.21.15.100:3301,v1:172.21.15.100:6790],172.21.15.122', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-13T11:52:13.144 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T11:52:13.144 DEBUG:teuthology.orchestra.run.smithi100:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T11:52:13.200 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.100 --addv c '[v2:172.21.15.100:3301,v1:172.21.15.100:6790]' --add b 172.21.15.122 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:52:13.286 INFO:teuthology.orchestra.run.smithi100.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:52:13.286 INFO:teuthology.orchestra.run.smithi100.stdout:monmaptool: generated fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:13.287 INFO:teuthology.orchestra.run.smithi100.stdout:setting min_mon_release = quincy 2024-09-13T11:52:13.287 INFO:teuthology.orchestra.run.smithi100.stdout:epoch 0 2024-09-13T11:52:13.287 INFO:teuthology.orchestra.run.smithi100.stdout:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:13.287 INFO:teuthology.orchestra.run.smithi100.stdout:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:13.287 INFO:teuthology.orchestra.run.smithi100.stdout:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:13.287 INFO:teuthology.orchestra.run.smithi100.stdout:min_mon_release 17 (quincy) 2024-09-13T11:52:13.287 INFO:teuthology.orchestra.run.smithi100.stdout:election_strategy: 1 2024-09-13T11:52:13.287 INFO:teuthology.orchestra.run.smithi100.stdout:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:13.287 INFO:teuthology.orchestra.run.smithi100.stdout:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:13.287 INFO:teuthology.orchestra.run.smithi100.stdout:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:13.287 INFO:teuthology.orchestra.run.smithi100.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-13T11:52:13.288 DEBUG:teuthology.orchestra.run.smithi100:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T11:52:13.303 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0... 2024-09-13T11:52:13.304 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T11:52:13.345 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T11:52:13.386 INFO:teuthology.orchestra.run.smithi100.stdout:[global] 2024-09-13T11:52:13.386 INFO:teuthology.orchestra.run.smithi100.stdout: chdir = "" 2024-09-13T11:52:13.386 INFO:teuthology.orchestra.run.smithi100.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T11:52:13.386 INFO:teuthology.orchestra.run.smithi100.stdout: auth supported = cephx 2024-09-13T11:52:13.386 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.386 INFO:teuthology.orchestra.run.smithi100.stdout: filestore xattr use omap = true 2024-09-13T11:52:13.386 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.386 INFO:teuthology.orchestra.run.smithi100.stdout: mon clock drift allowed = 1.000 2024-09-13T11:52:13.386 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.387 INFO:teuthology.orchestra.run.smithi100.stdout: osd crush chooseleaf type = 0 2024-09-13T11:52:13.387 INFO:teuthology.orchestra.run.smithi100.stdout: auth debug = true 2024-09-13T11:52:13.387 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.387 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on old message = true 2024-09-13T11:52:13.387 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on bug = true 2024-09-13T11:52:13.387 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.387 INFO:teuthology.orchestra.run.smithi100.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T11:52:13.387 INFO:teuthology.orchestra.run.smithi100.stdout: mon pg warn max object skew = 0 2024-09-13T11:52:13.387 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.387 INFO:teuthology.orchestra.run.smithi100.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T11:52:13.387 INFO:teuthology.orchestra.run.smithi100.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T11:52:13.387 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.387 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default size = 2 2024-09-13T11:52:13.388 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.388 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow primary affinity = true 2024-09-13T11:52:13.388 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow pg remap = true 2024-09-13T11:52:13.388 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on legacy crush tunables = false 2024-09-13T11:52:13.388 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on crush straw calc version zero = false 2024-09-13T11:52:13.388 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on no sortbitwise = false 2024-09-13T11:52:13.388 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on osd down out interval zero = false 2024-09-13T11:52:13.388 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on too few osds = false 2024-09-13T11:52:13.388 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T11:52:13.388 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T11:52:13.388 INFO:teuthology.orchestra.run.smithi100.stdout: mon_allow_pool_size_one = true 2024-09-13T11:52:13.388 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.389 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T11:52:13.389 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.389 INFO:teuthology.orchestra.run.smithi100.stdout: osd default data pool replay window = 5 2024-09-13T11:52:13.389 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.389 INFO:teuthology.orchestra.run.smithi100.stdout: mon allow pool delete = true 2024-09-13T11:52:13.389 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.389 INFO:teuthology.orchestra.run.smithi100.stdout: mon cluster log file level = debug 2024-09-13T11:52:13.389 INFO:teuthology.orchestra.run.smithi100.stdout: debug asserts on shutdown = true 2024-09-13T11:52:13.389 INFO:teuthology.orchestra.run.smithi100.stdout: mon health detail to clog = false 2024-09-13T11:52:13.389 INFO:teuthology.orchestra.run.smithi100.stdout: mon host = "172.21.15.100,[v2:172.21.15.100:3301,v1:172.21.15.100:6790],172.21.15.122" 2024-09-13T11:52:13.389 INFO:teuthology.orchestra.run.smithi100.stdout: mon client directed command retry = 5 2024-09-13T11:52:13.389 INFO:teuthology.orchestra.run.smithi100.stdout: mon election default strategy = 1 2024-09-13T11:52:13.389 INFO:teuthology.orchestra.run.smithi100.stdout: ms inject socket failures = 5000 2024-09-13T11:52:13.390 INFO:teuthology.orchestra.run.smithi100.stdout: ms type = async 2024-09-13T11:52:13.390 INFO:teuthology.orchestra.run.smithi100.stdout: fsid = 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:13.390 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.390 INFO:teuthology.orchestra.run.smithi100.stdout:[osd] 2024-09-13T11:52:13.390 INFO:teuthology.orchestra.run.smithi100.stdout: osd journal size = 100 2024-09-13T11:52:13.390 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.390 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub load threshold = 5.0 2024-09-13T11:52:13.390 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub max interval = 600 2024-09-13T11:52:13.390 INFO:teuthology.orchestra.run.smithi100.stdout: osd mclock profile = high_recovery_ops 2024-09-13T11:52:13.390 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.390 INFO:teuthology.orchestra.run.smithi100.stdout: osd recover clone overlap = true 2024-09-13T11:52:13.390 INFO:teuthology.orchestra.run.smithi100.stdout: osd recovery max chunk = 1048576 2024-09-13T11:52:13.390 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.391 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug shutdown = true 2024-09-13T11:52:13.391 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug op order = true 2024-09-13T11:52:13.391 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug verify stray on activate = true 2024-09-13T11:52:13.391 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug trim objects = true 2024-09-13T11:52:13.391 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.391 INFO:teuthology.orchestra.run.smithi100.stdout: osd open classes on start = true 2024-09-13T11:52:13.391 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug pg log writeout = true 2024-09-13T11:52:13.391 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.391 INFO:teuthology.orchestra.run.smithi100.stdout: osd deep scrub update digest min age = 30 2024-09-13T11:52:13.391 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.391 INFO:teuthology.orchestra.run.smithi100.stdout: osd map max advance = 10 2024-09-13T11:52:13.391 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.391 INFO:teuthology.orchestra.run.smithi100.stdout: journal zero on create = true 2024-09-13T11:52:13.391 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.392 INFO:teuthology.orchestra.run.smithi100.stdout: filestore ondisk finisher threads = 3 2024-09-13T11:52:13.392 INFO:teuthology.orchestra.run.smithi100.stdout: filestore apply finisher threads = 3 2024-09-13T11:52:13.392 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.392 INFO:teuthology.orchestra.run.smithi100.stdout: bdev debug aio = true 2024-09-13T11:52:13.392 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug misdirected ops = true 2024-09-13T11:52:13.392 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore block size = 96636764160 2024-09-13T11:52:13.392 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore compression algorithm = snappy 2024-09-13T11:52:13.392 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore compression mode = aggressive 2024-09-13T11:52:13.392 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore fsck on mount = True 2024-09-13T11:52:13.392 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore write v2 random = True 2024-09-13T11:52:13.392 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore zero block detection = True 2024-09-13T11:52:13.393 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluefs = 20 2024-09-13T11:52:13.393 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluestore = 20 2024-09-13T11:52:13.393 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-09-13T11:52:13.393 INFO:teuthology.orchestra.run.smithi100.stdout: debug osd = 20 2024-09-13T11:52:13.393 INFO:teuthology.orchestra.run.smithi100.stdout: debug rocksdb = 10 2024-09-13T11:52:13.393 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T11:52:13.393 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd full ratio = 0.9 2024-09-13T11:52:13.393 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd nearfull ratio = 0.8 2024-09-13T11:52:13.393 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug verify cached snaps = True 2024-09-13T11:52:13.393 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug verify missing on start = True 2024-09-13T11:52:13.393 INFO:teuthology.orchestra.run.smithi100.stdout: osd failsafe full ratio = 0.95 2024-09-13T11:52:13.393 INFO:teuthology.orchestra.run.smithi100.stdout: osd mclock override recovery settings = True 2024-09-13T11:52:13.393 INFO:teuthology.orchestra.run.smithi100.stdout: osd objectstore = bluestore 2024-09-13T11:52:13.394 INFO:teuthology.orchestra.run.smithi100.stdout: osd op queue = debug_random 2024-09-13T11:52:13.394 INFO:teuthology.orchestra.run.smithi100.stdout: osd op queue cut off = debug_random 2024-09-13T11:52:13.394 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.394 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr] 2024-09-13T11:52:13.394 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-09-13T11:52:13.394 INFO:teuthology.orchestra.run.smithi100.stdout: debug mgr = 20 2024-09-13T11:52:13.394 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2024-09-13T11:52:13.394 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2024-09-13T11:52:13.394 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2024-09-13T11:52:13.394 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2024-09-13T11:52:13.394 INFO:teuthology.orchestra.run.smithi100.stdout: mgr/telemetry/nag = false 2024-09-13T11:52:13.394 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.394 INFO:teuthology.orchestra.run.smithi100.stdout:[mon] 2024-09-13T11:52:13.395 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-09-13T11:52:13.395 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2024-09-13T11:52:13.395 INFO:teuthology.orchestra.run.smithi100.stdout: debug paxos = 20 2024-09-13T11:52:13.395 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2024-09-13T11:52:13.395 INFO:teuthology.orchestra.run.smithi100.stdout: mon data avail warn = 5 2024-09-13T11:52:13.395 INFO:teuthology.orchestra.run.smithi100.stdout: mon mgr mkfs grace = 240 2024-09-13T11:52:13.395 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2024-09-13T11:52:13.395 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd reporter subtree level = osd 2024-09-13T11:52:13.395 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd prime pg temp = true 2024-09-13T11:52:13.395 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2024-09-13T11:52:13.395 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.395 INFO:teuthology.orchestra.run.smithi100.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T11:52:13.395 INFO:teuthology.orchestra.run.smithi100.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T11:52:13.396 INFO:teuthology.orchestra.run.smithi100.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T11:52:13.396 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.396 INFO:teuthology.orchestra.run.smithi100.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T11:52:13.396 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T11:52:13.396 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T11:52:13.396 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.396 INFO:teuthology.orchestra.run.smithi100.stdout: # 1m isn't quite enough 2024-09-13T11:52:13.396 INFO:teuthology.orchestra.run.smithi100.stdout: mon_down_mkfs_grace = 2m 2024-09-13T11:52:13.396 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.396 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_filestore_osds = false 2024-09-13T11:52:13.396 INFO:teuthology.orchestra.run.smithi100.stdout: mon min osdmap epochs = 25 2024-09-13T11:52:13.396 INFO:teuthology.orchestra.run.smithi100.stdout: mon osdmap full prune interval = 2 2024-09-13T11:52:13.396 INFO:teuthology.orchestra.run.smithi100.stdout: mon osdmap full prune min = 15 2024-09-13T11:52:13.397 INFO:teuthology.orchestra.run.smithi100.stdout: mon osdmap full prune txsize = 2 2024-09-13T11:52:13.397 INFO:teuthology.orchestra.run.smithi100.stdout: mon scrub inject crc mismatch = 0.01 2024-09-13T11:52:13.397 INFO:teuthology.orchestra.run.smithi100.stdout: mon scrub inject missing keys = 0.05 2024-09-13T11:52:13.397 INFO:teuthology.orchestra.run.smithi100.stdout: mon scrub interval = 300 2024-09-13T11:52:13.397 INFO:teuthology.orchestra.run.smithi100.stdout: paxos service trim min = 5 2024-09-13T11:52:13.397 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:13.397 INFO:teuthology.orchestra.run.smithi100.stdout:[client] 2024-09-13T11:52:13.397 INFO:teuthology.orchestra.run.smithi100.stdout: rgw cache enabled = true 2024-09-13T11:52:13.397 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable ops log = true 2024-09-13T11:52:13.397 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable usage log = true 2024-09-13T11:52:13.397 INFO:teuthology.orchestra.run.smithi100.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T11:52:13.397 INFO:teuthology.orchestra.run.smithi100.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T11:52:13.397 INFO:teuthology.orchestra.run.smithi100.stdout: debug monc = 20 2024-09-13T11:52:13.398 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-09-13T11:52:13.398 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.a] 2024-09-13T11:52:13.398 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.c] 2024-09-13T11:52:13.398 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.b] 2024-09-13T11:52:13.398 INFO:teuthology.orchestra.run.smithi122.stdout:[global] 2024-09-13T11:52:13.398 INFO:teuthology.orchestra.run.smithi122.stdout: chdir = "" 2024-09-13T11:52:13.398 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T11:52:13.399 INFO:teuthology.orchestra.run.smithi122.stdout: auth supported = cephx 2024-09-13T11:52:13.399 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.399 INFO:teuthology.orchestra.run.smithi122.stdout: filestore xattr use omap = true 2024-09-13T11:52:13.399 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.399 INFO:teuthology.orchestra.run.smithi122.stdout: mon clock drift allowed = 1.000 2024-09-13T11:52:13.399 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.399 INFO:teuthology.orchestra.run.smithi122.stdout: osd crush chooseleaf type = 0 2024-09-13T11:52:13.399 INFO:teuthology.orchestra.run.smithi122.stdout: auth debug = true 2024-09-13T11:52:13.399 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.400 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on old message = true 2024-09-13T11:52:13.400 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on bug = true 2024-09-13T11:52:13.400 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.400 INFO:teuthology.orchestra.run.smithi122.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T11:52:13.400 INFO:teuthology.orchestra.run.smithi122.stdout: mon pg warn max object skew = 0 2024-09-13T11:52:13.400 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.400 INFO:teuthology.orchestra.run.smithi122.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T11:52:13.400 INFO:teuthology.orchestra.run.smithi122.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T11:52:13.401 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.401 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default size = 2 2024-09-13T11:52:13.401 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.401 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow primary affinity = true 2024-09-13T11:52:13.401 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow pg remap = true 2024-09-13T11:52:13.401 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on legacy crush tunables = false 2024-09-13T11:52:13.401 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on crush straw calc version zero = false 2024-09-13T11:52:13.401 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on no sortbitwise = false 2024-09-13T11:52:13.401 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on osd down out interval zero = false 2024-09-13T11:52:13.401 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on too few osds = false 2024-09-13T11:52:13.401 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T11:52:13.401 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T11:52:13.401 INFO:teuthology.orchestra.run.smithi122.stdout: mon_allow_pool_size_one = true 2024-09-13T11:52:13.402 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.402 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T11:52:13.402 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.402 INFO:teuthology.orchestra.run.smithi122.stdout: osd default data pool replay window = 5 2024-09-13T11:52:13.402 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.402 INFO:teuthology.orchestra.run.smithi122.stdout: mon allow pool delete = true 2024-09-13T11:52:13.402 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.402 INFO:teuthology.orchestra.run.smithi122.stdout: mon cluster log file level = debug 2024-09-13T11:52:13.402 INFO:teuthology.orchestra.run.smithi122.stdout: debug asserts on shutdown = true 2024-09-13T11:52:13.402 INFO:teuthology.orchestra.run.smithi122.stdout: mon health detail to clog = false 2024-09-13T11:52:13.403 INFO:teuthology.orchestra.run.smithi122.stdout: mon host = "172.21.15.100,[v2:172.21.15.100:3301,v1:172.21.15.100:6790],172.21.15.122" 2024-09-13T11:52:13.403 INFO:teuthology.orchestra.run.smithi122.stdout: mon client directed command retry = 5 2024-09-13T11:52:13.403 INFO:teuthology.orchestra.run.smithi122.stdout: mon election default strategy = 1 2024-09-13T11:52:13.403 INFO:teuthology.orchestra.run.smithi122.stdout: ms inject socket failures = 5000 2024-09-13T11:52:13.403 INFO:teuthology.orchestra.run.smithi122.stdout: ms type = async 2024-09-13T11:52:13.403 INFO:teuthology.orchestra.run.smithi122.stdout: fsid = 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:13.403 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.403 INFO:teuthology.orchestra.run.smithi122.stdout:[osd] 2024-09-13T11:52:13.403 INFO:teuthology.orchestra.run.smithi122.stdout: osd journal size = 100 2024-09-13T11:52:13.403 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.403 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub load threshold = 5.0 2024-09-13T11:52:13.404 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub max interval = 600 2024-09-13T11:52:13.404 INFO:teuthology.orchestra.run.smithi122.stdout: osd mclock profile = high_recovery_ops 2024-09-13T11:52:13.404 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.404 INFO:teuthology.orchestra.run.smithi122.stdout: osd recover clone overlap = true 2024-09-13T11:52:13.404 INFO:teuthology.orchestra.run.smithi122.stdout: osd recovery max chunk = 1048576 2024-09-13T11:52:13.404 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.404 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug shutdown = true 2024-09-13T11:52:13.404 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug op order = true 2024-09-13T11:52:13.404 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify stray on activate = true 2024-09-13T11:52:13.404 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug trim objects = true 2024-09-13T11:52:13.404 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.404 INFO:teuthology.orchestra.run.smithi122.stdout: osd open classes on start = true 2024-09-13T11:52:13.404 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug pg log writeout = true 2024-09-13T11:52:13.405 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.405 INFO:teuthology.orchestra.run.smithi122.stdout: osd deep scrub update digest min age = 30 2024-09-13T11:52:13.405 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.405 INFO:teuthology.orchestra.run.smithi122.stdout: osd map max advance = 10 2024-09-13T11:52:13.405 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.405 INFO:teuthology.orchestra.run.smithi122.stdout: journal zero on create = true 2024-09-13T11:52:13.405 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.405 INFO:teuthology.orchestra.run.smithi122.stdout: filestore ondisk finisher threads = 3 2024-09-13T11:52:13.405 INFO:teuthology.orchestra.run.smithi122.stdout: filestore apply finisher threads = 3 2024-09-13T11:52:13.405 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.405 INFO:teuthology.orchestra.run.smithi122.stdout: bdev debug aio = true 2024-09-13T11:52:13.405 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug misdirected ops = true 2024-09-13T11:52:13.405 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore block size = 96636764160 2024-09-13T11:52:13.406 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore compression algorithm = snappy 2024-09-13T11:52:13.406 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore compression mode = aggressive 2024-09-13T11:52:13.406 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore fsck on mount = True 2024-09-13T11:52:13.406 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore write v2 random = True 2024-09-13T11:52:13.406 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore zero block detection = True 2024-09-13T11:52:13.406 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluefs = 20 2024-09-13T11:52:13.406 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluestore = 20 2024-09-13T11:52:13.406 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-09-13T11:52:13.406 INFO:teuthology.orchestra.run.smithi122.stdout: debug osd = 20 2024-09-13T11:52:13.406 INFO:teuthology.orchestra.run.smithi122.stdout: debug rocksdb = 10 2024-09-13T11:52:13.406 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T11:52:13.406 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd full ratio = 0.9 2024-09-13T11:52:13.406 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd nearfull ratio = 0.8 2024-09-13T11:52:13.407 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify cached snaps = True 2024-09-13T11:52:13.407 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify missing on start = True 2024-09-13T11:52:13.407 INFO:teuthology.orchestra.run.smithi122.stdout: osd failsafe full ratio = 0.95 2024-09-13T11:52:13.407 INFO:teuthology.orchestra.run.smithi122.stdout: osd mclock override recovery settings = True 2024-09-13T11:52:13.407 INFO:teuthology.orchestra.run.smithi122.stdout: osd objectstore = bluestore 2024-09-13T11:52:13.407 INFO:teuthology.orchestra.run.smithi122.stdout: osd op queue = debug_random 2024-09-13T11:52:13.407 INFO:teuthology.orchestra.run.smithi122.stdout: osd op queue cut off = debug_random 2024-09-13T11:52:13.407 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.407 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr] 2024-09-13T11:52:13.407 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-09-13T11:52:13.407 INFO:teuthology.orchestra.run.smithi122.stdout: debug mgr = 20 2024-09-13T11:52:13.407 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-09-13T11:52:13.408 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-09-13T11:52:13.408 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-09-13T11:52:13.408 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-09-13T11:52:13.408 INFO:teuthology.orchestra.run.smithi122.stdout: mgr/telemetry/nag = false 2024-09-13T11:52:13.408 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.408 INFO:teuthology.orchestra.run.smithi122.stdout:[mon] 2024-09-13T11:52:13.408 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-09-13T11:52:13.408 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-09-13T11:52:13.408 INFO:teuthology.orchestra.run.smithi122.stdout: debug paxos = 20 2024-09-13T11:52:13.408 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-09-13T11:52:13.408 INFO:teuthology.orchestra.run.smithi122.stdout: mon data avail warn = 5 2024-09-13T11:52:13.408 INFO:teuthology.orchestra.run.smithi122.stdout: mon mgr mkfs grace = 240 2024-09-13T11:52:13.408 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-09-13T11:52:13.409 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd reporter subtree level = osd 2024-09-13T11:52:13.409 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd prime pg temp = true 2024-09-13T11:52:13.409 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-09-13T11:52:13.409 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.409 INFO:teuthology.orchestra.run.smithi122.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T11:52:13.409 INFO:teuthology.orchestra.run.smithi122.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T11:52:13.409 INFO:teuthology.orchestra.run.smithi122.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T11:52:13.409 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.409 INFO:teuthology.orchestra.run.smithi122.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T11:52:13.409 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T11:52:13.409 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T11:52:13.409 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.409 INFO:teuthology.orchestra.run.smithi122.stdout: # 1m isn't quite enough 2024-09-13T11:52:13.409 INFO:teuthology.orchestra.run.smithi122.stdout: mon_down_mkfs_grace = 2m 2024-09-13T11:52:13.410 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.410 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_filestore_osds = false 2024-09-13T11:52:13.410 INFO:teuthology.orchestra.run.smithi122.stdout: mon min osdmap epochs = 25 2024-09-13T11:52:13.410 INFO:teuthology.orchestra.run.smithi122.stdout: mon osdmap full prune interval = 2 2024-09-13T11:52:13.410 INFO:teuthology.orchestra.run.smithi122.stdout: mon osdmap full prune min = 15 2024-09-13T11:52:13.410 INFO:teuthology.orchestra.run.smithi122.stdout: mon osdmap full prune txsize = 2 2024-09-13T11:52:13.410 INFO:teuthology.orchestra.run.smithi122.stdout: mon scrub inject crc mismatch = 0.01 2024-09-13T11:52:13.410 INFO:teuthology.orchestra.run.smithi122.stdout: mon scrub inject missing keys = 0.05 2024-09-13T11:52:13.410 INFO:teuthology.orchestra.run.smithi122.stdout: mon scrub interval = 300 2024-09-13T11:52:13.410 INFO:teuthology.orchestra.run.smithi122.stdout: paxos service trim min = 5 2024-09-13T11:52:13.410 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T11:52:13.410 INFO:teuthology.orchestra.run.smithi122.stdout:[client] 2024-09-13T11:52:13.410 INFO:teuthology.orchestra.run.smithi122.stdout: rgw cache enabled = true 2024-09-13T11:52:13.410 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable ops log = true 2024-09-13T11:52:13.411 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable usage log = true 2024-09-13T11:52:13.411 INFO:teuthology.orchestra.run.smithi122.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T11:52:13.411 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T11:52:13.411 INFO:teuthology.orchestra.run.smithi122.stdout: debug monc = 20 2024-09-13T11:52:13.411 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-09-13T11:52:13.411 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.a] 2024-09-13T11:52:13.411 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.c] 2024-09-13T11:52:13.411 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.b] 2024-09-13T11:52:13.412 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-13T11:52:13.413 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-13T11:52:13.507 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-13T11:52:13.507 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T11:52:13.507 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-13T11:52:13.522 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T11:52:13.522 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-13T11:52:13.577 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi100.front.sepia.ceph.com 2024-09-13T11:52:13.577 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T11:52:13.577 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T11:52:13.577 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T11:52:13.651 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T11:52:13.651 DEBUG:teuthology.orchestra.run.smithi100:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:52:13.706 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T11:52:13.707 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T11:52:13.707 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T11:52:13.707 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T11:52:13.742 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T11:52:13.742 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:52:13.797 INFO:tasks.ceph:Setting up mon nodes... 2024-09-13T11:52:13.797 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-13T11:52:13.798 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T11:52:13.899 INFO:teuthology.orchestra.run.smithi122.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T11:52:13.901 INFO:tasks.ceph:Setting up mds nodes... 2024-09-13T11:52:13.901 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-13T11:52:13.902 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-13T11:52:13.943 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-13T11:52:13.956 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-13T11:52:13.956 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi100.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi122.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-13T11:52:13.956 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-13T11:52:13.981 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T11:52:13.981 INFO:tasks.ceph:role: osd.0 2024-09-13T11:52:13.982 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-13T11:52:13.982 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T11:52:14.053 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:52:14.053 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:52:14.053 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:52:14.054 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T11:52:14.054 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:52:14.054 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:52:14.054 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:52:14.054 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T11:52:14.054 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:52:14.054 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:52:14.555 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2024-09-13T11:52:14.558 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2024-09-13T11:52:14.558 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-13T11:52:14.681 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-13T11:52:14.713 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-13T11:52:14.778 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T11:52:14.778 INFO:tasks.ceph:role: osd.1 2024-09-13T11:52:14.778 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-13T11:52:14.779 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T11:52:14.850 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:52:14.851 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:52:14.851 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:52:14.851 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T11:52:14.851 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:52:14.851 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:52:14.851 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:52:14.851 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T11:52:14.851 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:52:14.852 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:52:15.315 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2024-09-13T11:52:15.318 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2024-09-13T11:52:15.318 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-13T11:52:15.411 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-13T11:52:15.442 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-13T11:52:15.509 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T11:52:15.509 INFO:tasks.ceph:role: osd.2 2024-09-13T11:52:15.509 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-13T11:52:15.510 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T11:52:15.582 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:52:15.582 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:52:15.583 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:52:15.583 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T11:52:15.583 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:52:15.583 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:52:15.583 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:52:15.583 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T11:52:15.584 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:52:15.584 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:52:16.076 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2024-09-13T11:52:16.078 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2024-09-13T11:52:16.079 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-13T11:52:16.212 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-13T11:52:16.244 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:52:16.352 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:16.351+0000 7fa0210a5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-13T11:52:16.356 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:16.354+0000 7fa0210a5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-13T11:52:16.356 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:16.354+0000 7fa0210a5640 -1 bdev(0x55c20f497800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-13T11:52:16.356 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:16.354+0000 7fa0210a5640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-13T11:52:19.515 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T11:52:19.545 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:52:19.638 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:19.636+0000 7fadda551640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-13T11:52:19.638 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:19.637+0000 7fadda551640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-13T11:52:19.639 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:19.637+0000 7fadda551640 -1 bdev(0x5609c4fc9800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-13T11:52:19.639 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:19.637+0000 7fadda551640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-13T11:52:23.301 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T11:52:23.330 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:52:23.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:23.419+0000 7fb3d5f20640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-13T11:52:23.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:23.419+0000 7fb3d5f20640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-13T11:52:23.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:23.419+0000 7fb3d5f20640 -1 bdev(0x556425139800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-13T11:52:23.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:23.419+0000 7fb3d5f20640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-13T11:52:27.079 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T11:52:27.109 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-13T11:52:27.139 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-13T11:52:27.139 INFO:tasks.ceph:role: osd.3 2024-09-13T11:52:27.139 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T11:52:27.139 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T11:52:27.215 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:52:27.216 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:52:27.216 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:52:27.216 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T11:52:27.216 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:52:27.216 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:52:27.216 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:52:27.216 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T11:52:27.216 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:52:27.216 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:52:27.723 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-09-13T11:52:27.725 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-09-13T11:52:27.725 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-13T11:52:27.863 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-13T11:52:27.895 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-13T11:52:27.960 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-13T11:52:27.961 INFO:tasks.ceph:role: osd.4 2024-09-13T11:52:27.961 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T11:52:27.961 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T11:52:28.033 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:52:28.033 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:52:28.034 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:52:28.034 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T11:52:28.034 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:52:28.034 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:52:28.034 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:52:28.034 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T11:52:28.034 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:52:28.034 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:52:28.478 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-09-13T11:52:28.480 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-09-13T11:52:28.481 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-13T11:52:28.562 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-13T11:52:28.630 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-13T11:52:28.695 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-13T11:52:28.695 INFO:tasks.ceph:role: osd.5 2024-09-13T11:52:28.695 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T11:52:28.696 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T11:52:28.767 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:52:28.768 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:52:28.768 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:52:28.768 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T11:52:28.768 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:52:28.768 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:52:28.768 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:52:28.768 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T11:52:28.768 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:52:28.768 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:52:29.248 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-09-13T11:52:29.250 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-09-13T11:52:29.250 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-13T11:52:29.382 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-13T11:52:29.414 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:52:29.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:29.504+0000 7f10a70f1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-13T11:52:29.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:29.509+0000 7f10a70f1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-13T11:52:29.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:29.509+0000 7f10a70f1640 -1 bdev(0x562a0fead800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-13T11:52:29.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:29.510+0000 7f10a70f1640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-13T11:52:32.929 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T11:52:32.959 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:52:33.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:33.048+0000 7fd9a721d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-13T11:52:33.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:33.048+0000 7fd9a721d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-13T11:52:33.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:33.048+0000 7fd9a721d640 -1 bdev(0x55f84d4e7800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-13T11:52:33.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:33.048+0000 7fd9a721d640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-13T11:52:36.210 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-13T11:52:36.240 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:52:36.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:36.331+0000 7f55629bb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-13T11:52:36.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:36.332+0000 7f55629bb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-13T11:52:36.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:36.332+0000 7f55629bb640 -1 bdev(0x55a995e6d800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-13T11:52:36.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:36.332+0000 7f55629bb640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-13T11:52:39.248 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-13T11:52:39.275 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-13T11:52:39.275 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T11:52:39.275 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-13T11:52:39.305 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T11:52:39.305 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-13T11:52:39.375 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T11:52:39.375 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-13T11:52:39.444 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T11:52:39.444 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-13T11:52:39.471 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T11:52:39.471 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-13T11:52:39.538 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T11:52:39.538 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-13T11:52:39.604 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T11:52:39.605 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-13T11:52:39.670 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T11:52:39.670 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-13T11:52:39.725 INFO:tasks.ceph:Adding keys to all mons... 2024-09-13T11:52:39.725 DEBUG:teuthology.orchestra.run.smithi100:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T11:52:39.728 DEBUG:teuthology.orchestra.run.smithi122:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T11:52:39.769 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.0] 2024-09-13T11:52:39.769 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQBwJ+RmOE/9FBAA0COC2An5sJa26uOiJUZXLg== 2024-09-13T11:52:39.769 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.1] 2024-09-13T11:52:39.769 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQBzJ+Rm2YMDJhAA8FahmHHQuJ0t96uroAPzzQ== 2024-09-13T11:52:39.769 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.2] 2024-09-13T11:52:39.769 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQB3J+Rm+JYKGRAAn5TTvUSAqkegbC9lhfMOdQ== 2024-09-13T11:52:39.769 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr.x] 2024-09-13T11:52:39.769 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQBtJ+RmlBCNNRAAK2cdMWrJuyNBGpKeSpSMvg== 2024-09-13T11:52:39.770 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.3] 2024-09-13T11:52:39.770 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQB9J+RmtU0aHhAAoNjXgMJzAVL352Ur/Ppwtg== 2024-09-13T11:52:39.770 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.4] 2024-09-13T11:52:39.770 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCBJ+RmXKP0AhAAoJExghw02gpNO7/qHxPgtg== 2024-09-13T11:52:39.770 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.5] 2024-09-13T11:52:39.770 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCEJ+RmBVHdExAAo+Qvrt1mhsFW5XsgKlw/pA== 2024-09-13T11:52:39.770 INFO:teuthology.orchestra.run.smithi100.stdout:[client.0] 2024-09-13T11:52:39.770 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQBtJ+Rmepk3OBAAfJoaGGPKjSrYVZMgkVMmoQ== 2024-09-13T11:52:39.789 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.0] 2024-09-13T11:52:39.789 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBwJ+RmOE/9FBAA0COC2An5sJa26uOiJUZXLg== 2024-09-13T11:52:39.790 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.1] 2024-09-13T11:52:39.790 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBzJ+Rm2YMDJhAA8FahmHHQuJ0t96uroAPzzQ== 2024-09-13T11:52:39.790 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.2] 2024-09-13T11:52:39.790 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQB3J+Rm+JYKGRAAn5TTvUSAqkegbC9lhfMOdQ== 2024-09-13T11:52:39.790 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr.x] 2024-09-13T11:52:39.790 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBtJ+RmlBCNNRAAK2cdMWrJuyNBGpKeSpSMvg== 2024-09-13T11:52:39.791 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.3] 2024-09-13T11:52:39.791 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQB9J+RmtU0aHhAAoNjXgMJzAVL352Ur/Ppwtg== 2024-09-13T11:52:39.791 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.4] 2024-09-13T11:52:39.791 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCBJ+RmXKP0AhAAoJExghw02gpNO7/qHxPgtg== 2024-09-13T11:52:39.791 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.5] 2024-09-13T11:52:39.791 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCEJ+RmBVHdExAAo+Qvrt1mhsFW5XsgKlw/pA== 2024-09-13T11:52:39.792 INFO:teuthology.orchestra.run.smithi122.stdout:[client.0] 2024-09-13T11:52:39.792 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBtJ+Rmepk3OBAAfJoaGGPKjSrYVZMgkVMmoQ== 2024-09-13T11:52:39.793 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:52:39.814 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:52:39.885 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:52:39.887 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:52:39.938 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:52:39.941 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:52:39.991 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T11:52:39.993 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T11:52:40.044 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:52:40.047 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:52:40.097 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:52:40.099 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:52:40.151 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:52:40.153 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:52:40.203 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T11:52:40.206 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T11:52:40.259 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-13T11:52:40.259 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-13T11:52:40.284 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T11:52:40.414 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-13T11:52:40.439 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-13T11:52:40.514 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T11:52:40.619 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-13T11:52:40.644 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-13T11:52:40.669 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T11:52:40.769 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-13T11:52:40.796 DEBUG:teuthology.orchestra.run.smithi100:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:52:40.799 DEBUG:teuthology.orchestra.run.smithi122:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:52:40.851 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-13T11:52:40.851 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-13T11:52:40.851 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-13T11:52:40.854 INFO:tasks.ceph.mon.a:Started 2024-09-13T11:52:40.854 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-13T11:52:40.855 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-13T11:52:40.857 INFO:tasks.ceph.mon.c:Started 2024-09-13T11:52:40.857 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-13T11:52:40.857 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-13T11:52:40.894 INFO:tasks.ceph.mon.b:Started 2024-09-13T11:52:40.895 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-13T11:52:40.895 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-13T11:52:40.895 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-13T11:52:40.897 INFO:tasks.ceph.mgr.x:Started 2024-09-13T11:52:40.897 DEBUG:tasks.ceph:set 0 configs 2024-09-13T11:52:40.897 INFO:tasks.ceph:Setting crush tunables to default 2024-09-13T11:52:40.897 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd crush tunables default 2024-09-13T11:52:40.997 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.995+0000 7f739495d640 10 monclient: get_monmap_and_config 2024-09-13T11:52:40.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.995+0000 7f739495d640 10 monclient: build_initial_monmap 2024-09-13T11:52:40.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.995+0000 7f739495d640 10 monclient: monmap: 2024-09-13T11:52:40.998 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:52:40.998 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:40.998 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:40.996518+0000 2024-09-13T11:52:40.998 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:40.996518+0000 2024-09-13T11:52:40.998 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:52:40.999 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:40.999 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:52:40.999 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:52:40.999 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:52:40.999 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:52:40.999 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:52:40.999 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:52:40.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.996+0000 7f739495d640 1 Processor -- start 2024-09-13T11:52:41.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.996+0000 7f739495d640 1 -- start start 2024-09-13T11:52:41.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.996+0000 7f739495d640 10 monclient: init 2024-09-13T11:52:41.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.996+0000 7f739495d640 10 monclient: _reopen_session rank -1 2024-09-13T11:52:41.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.996+0000 7f739495d640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-13T11:52:41.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.996+0000 7f739495d640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f739006ea50 0x7f73900a2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:41.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.996+0000 7f739495d640 10 monclient(hunting): picked mon.noname-b con 0x7f739006ea50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:52:41.003 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.996+0000 7f739495d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f739006ee30 addr v1:172.21.15.122:6789/0 2024-09-13T11:52:41.003 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.996+0000 7f739495d640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f739013caf0 0x7f739013eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:41.003 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.996+0000 7f739495d640 10 monclient(hunting): picked mon.noname-a con 0x7f739013caf0 addr v2:172.21.15.100:3300/0 2024-09-13T11:52:41.003 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.996+0000 7f739495d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73900a6440 con 0x7f739006ee30 2024-09-13T11:52:41.003 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.996+0000 7f739495d640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:41.004 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.996+0000 7f739495d640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f73900a1e50 con 0x7f739013caf0 2024-09-13T11:52:41.004 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.996+0000 7f739495d640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:41.004 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.996+0000 7f739495d640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f73900a6680 con 0x7f739006ea50 2024-09-13T11:52:41.004 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.996+0000 7f739495d640 10 monclient(hunting): _renew_subs 2024-09-13T11:52:41.004 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.996+0000 7f739495d640 10 monclient(hunting): authenticate will time out at 741.589981s 2024-09-13T11:52:41.004 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.997+0000 7f738e59c640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f739006ea50 0x7f73900a2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:41.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.997+0000 7f738ed9d640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f739013caf0 0x7f739013eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:41.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.997+0000 7f738e59c640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f739006ea50 0x7f73900a2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:49754/0 (socket says 172.21.15.100:49754) 2024-09-13T11:52:41.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.997+0000 7f738e59c640 1 -- 172.21.15.100:0/3886219220 learned_addr learned my addr 172.21.15.100:0/3886219220 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:52:41.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.997+0000 7f738ed9d640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f739013caf0 0x7f739013eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:48580/0 (socket says 172.21.15.100:48580) 2024-09-13T11:52:41.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.997+0000 7f738e59c640 10 monclient(hunting): get_auth_request con 0x7f739006ea50 auth_method 0 2024-09-13T11:52:41.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.997+0000 7f738e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:41.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.997+0000 7f738e59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:41.006 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.997+0000 7f738e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:41.006 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.997+0000 7f738ed9d640 10 monclient(hunting): get_auth_request con 0x7f739013caf0 auth_method 0 2024-09-13T11:52:41.006 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.997+0000 7f738ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:41.006 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.997+0000 7f738ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:41.006 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.997+0000 7f738ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:41.006 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.997+0000 7f738e59c640 1 -- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f739006ea50 msgr2=0x7f73900a2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T11:52:41.006 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.997+0000 7f738e59c640 1 -- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f739006ea50 msgr2=0x7f73900a2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:52:41.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.997+0000 7f738ed9d640 1 -- 172.21.15.100:0/3886219220 >> v2:172.21.15.100:3300/0 conn(0x7f739013caf0 msgr2=0x7f739013eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-13T11:52:41.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.997+0000 7f738ed9d640 1 -- 172.21.15.100:0/3886219220 >> v2:172.21.15.100:3300/0 conn(0x7f739013caf0 msgr2=0x7f739013eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:52:41.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.997+0000 7f738e59c640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f739006ea50 0x7f73900a2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:52:41.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.997+0000 7f738ed9d640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.100:3300/0 conn(0x7f739013caf0 0x7f739013eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:52:41.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.997+0000 7f738e59c640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f739006ea50 0x7f73900a2830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:52:41.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:40.997+0000 7f738ed9d640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.100:3300/0 conn(0x7f739013caf0 0x7f739013eee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:52:41.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.198+0000 7f738ed9d640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.100:3300/0 conn(0x7f739013caf0 0x7f739013eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:41.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.198+0000 7f738e59c640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f739006ea50 0x7f73900a2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:41.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.198+0000 7f738ed9d640 10 monclient(hunting): get_auth_request con 0x7f739013caf0 auth_method 0 2024-09-13T11:52:41.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.198+0000 7f738ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:41.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.198+0000 7f738ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:41.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.198+0000 7f738ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:52:41.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.198+0000 7f738e59c640 10 monclient(hunting): get_auth_request con 0x7f739006ea50 auth_method 0 2024-09-13T11:52:41.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.198+0000 7f738e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:41.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.198+0000 7f738e59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:41.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.198+0000 7f738e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:52:41.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.198+0000 7f738ed9d640 1 -- 172.21.15.100:0/3886219220 >> v2:172.21.15.100:3300/0 conn(0x7f739013caf0 msgr2=0x7f739013eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T11:52:41.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.198+0000 7f738ed9d640 1 -- 172.21.15.100:0/3886219220 >> v2:172.21.15.100:3300/0 conn(0x7f739013caf0 msgr2=0x7f739013eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:52:41.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.198+0000 7f738ed9d640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.100:3300/0 conn(0x7f739013caf0 0x7f739013eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:52:41.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.198+0000 7f738e59c640 1 -- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f739006ea50 msgr2=0x7f73900a2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-13T11:52:41.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.198+0000 7f738e59c640 1 -- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f739006ea50 msgr2=0x7f73900a2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:52:41.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.198+0000 7f738e59c640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f739006ea50 0x7f73900a2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:52:41.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.198+0000 7f738ed9d640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.100:3300/0 conn(0x7f739013caf0 0x7f739013eee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T11:52:41.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.198+0000 7f738e59c640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f739006ea50 0x7f73900a2830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T11:52:41.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.599+0000 7f738ed9d640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.100:3300/0 conn(0x7f739013caf0 0x7f739013eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:41.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.599+0000 7f738e59c640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f739006ea50 0x7f73900a2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:41.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.599+0000 7f738ed9d640 10 monclient(hunting): get_auth_request con 0x7f739013caf0 auth_method 0 2024-09-13T11:52:41.602 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.599+0000 7f738ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:41.602 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.599+0000 7f738ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:41.602 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.599+0000 7f738ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:52:41.602 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.599+0000 7f738e59c640 10 monclient(hunting): get_auth_request con 0x7f739006ea50 auth_method 0 2024-09-13T11:52:41.602 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.599+0000 7f738e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:41.602 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.599+0000 7f738e59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:41.602 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.599+0000 7f738e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:52:41.602 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.599+0000 7f738ed9d640 1 -- 172.21.15.100:0/3886219220 >> v2:172.21.15.100:3300/0 conn(0x7f739013caf0 msgr2=0x7f739013eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-13T11:52:41.602 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.599+0000 7f738ed9d640 1 -- 172.21.15.100:0/3886219220 >> v2:172.21.15.100:3300/0 conn(0x7f739013caf0 msgr2=0x7f739013eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:52:41.602 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.599+0000 7f738ed9d640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.100:3300/0 conn(0x7f739013caf0 0x7f739013eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:52:41.602 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.599+0000 7f738e59c640 1 -- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f739006ea50 msgr2=0x7f73900a2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T11:52:41.602 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.599+0000 7f738e59c640 1 -- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f739006ea50 msgr2=0x7f73900a2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:52:41.602 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.599+0000 7f738e59c640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f739006ea50 0x7f73900a2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:52:41.603 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.599+0000 7f738ed9d640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.100:3300/0 conn(0x7f739013caf0 0x7f739013eee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T11:52:41.603 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:41.599+0000 7f738e59c640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f739006ea50 0x7f73900a2830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T11:52:42.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:42.400+0000 7f738e59c640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f739006ea50 0x7f73900a2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:42.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:42.400+0000 7f738ed9d640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.100:3300/0 conn(0x7f739013caf0 0x7f739013eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:42.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:42.400+0000 7f738e59c640 10 monclient(hunting): get_auth_request con 0x7f739006ea50 auth_method 0 2024-09-13T11:52:42.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:42.400+0000 7f738e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:42.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:42.400+0000 7f738e59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:42.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:42.400+0000 7f738e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:52:42.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:42.401+0000 7f738ed9d640 10 monclient(hunting): get_auth_request con 0x7f739013caf0 auth_method 0 2024-09-13T11:52:42.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:42.401+0000 7f738ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:42.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:42.401+0000 7f738ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:42.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:42.401+0000 7f738ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:52:42.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:42.401+0000 7f738e59c640 1 -- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f739006ea50 msgr2=0x7f73900a2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T11:52:42.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:42.401+0000 7f738e59c640 1 -- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f739006ea50 msgr2=0x7f73900a2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:52:42.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:42.401+0000 7f738e59c640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f739006ea50 0x7f73900a2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:52:42.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:42.401+0000 7f738ed9d640 1 -- 172.21.15.100:0/3886219220 >> v2:172.21.15.100:3300/0 conn(0x7f739013caf0 msgr2=0x7f739013eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-13T11:52:42.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:42.401+0000 7f738ed9d640 1 -- 172.21.15.100:0/3886219220 >> v2:172.21.15.100:3300/0 conn(0x7f739013caf0 msgr2=0x7f739013eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:52:42.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:42.401+0000 7f738ed9d640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.100:3300/0 conn(0x7f739013caf0 0x7f739013eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:52:42.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:42.401+0000 7f738e59c640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f739006ea50 0x7f73900a2830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T11:52:42.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:42.401+0000 7f738ed9d640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.100:3300/0 conn(0x7f739013caf0 0x7f739013eee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T11:52:43.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f737ffff640 10 monclient(hunting): tick 2024-09-13T11:52:43.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f737ffff640 10 monclient(hunting): _check_auth_tickets 2024-09-13T11:52:43.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f737ffff640 1 monclient(hunting): continuing hunt 2024-09-13T11:52:43.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f737ffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-13T11:52:43.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f737ffff640 1 -- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f739006ea50 msgr2=0x7f73900a2830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:52:43.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f737ffff640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f739006ea50 0x7f73900a2830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:43.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f737ffff640 1 -- 172.21.15.100:0/3886219220 >> v2:172.21.15.100:3300/0 conn(0x7f739013caf0 msgr2=0x7f739013eee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:52:43.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f737ffff640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.100:3300/0 conn(0x7f739013caf0 0x7f739013eee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:44.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f737ffff640 1 -- 172.21.15.100:0/3886219220 >> v1:172.21.15.122:6789/0 conn(0x7f739006ee30 legacy=0x7f739006f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:44.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f737ffff640 10 monclient: _add_conns ranks=[0,4] 2024-09-13T11:52:44.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f737ffff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f73700036e0 addr v1:172.21.15.100:6789/0 2024-09-13T11:52:44.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f737ffff640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.122:3300/0 conn(0x7f7370007430 0x7f73700098b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:44.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f737ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f7370007430 addr v2:172.21.15.122:3300/0 2024-09-13T11:52:44.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f737ffff640 1 -- 172.21.15.100:0/3886219220 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f739006fc60 con 0x7f73700036e0 2024-09-13T11:52:44.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f737ffff640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:44.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f737ffff640 1 -- 172.21.15.100:0/3886219220 --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f73900a1e50 con 0x7f7370007430 2024-09-13T11:52:44.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f737ffff640 10 monclient(hunting): _renew_subs 2024-09-13T11:52:44.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f738dd9b640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.122:3300/0 conn(0x7f7370007430 0x7f73700098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:44.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f738dd9b640 10 monclient(hunting): get_auth_request con 0x7f7370007430 auth_method 0 2024-09-13T11:52:44.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f738dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:44.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f738dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:44.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f738dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:44.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f738dd9b640 1 -- 172.21.15.100:0/3886219220 >> v2:172.21.15.122:3300/0 conn(0x7f7370007430 msgr2=0x7f73700098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T11:52:44.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f738dd9b640 1 -- 172.21.15.100:0/3886219220 >> v2:172.21.15.122:3300/0 conn(0x7f7370007430 msgr2=0x7f73700098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:52:44.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f738dd9b640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.122:3300/0 conn(0x7f7370007430 0x7f73700098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:52:44.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:43.997+0000 7f738dd9b640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.122:3300/0 conn(0x7f7370007430 0x7f73700098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:52:44.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:44.198+0000 7f738dd9b640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.122:3300/0 conn(0x7f7370007430 0x7f73700098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:44.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:44.198+0000 7f738dd9b640 10 monclient(hunting): get_auth_request con 0x7f7370007430 auth_method 0 2024-09-13T11:52:44.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:44.198+0000 7f738dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:44.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:44.198+0000 7f738dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:44.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:44.198+0000 7f738dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:52:44.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:44.198+0000 7f738dd9b640 1 -- 172.21.15.100:0/3886219220 >> v2:172.21.15.122:3300/0 conn(0x7f7370007430 msgr2=0x7f73700098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T11:52:44.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:44.198+0000 7f738dd9b640 1 -- 172.21.15.100:0/3886219220 >> v2:172.21.15.122:3300/0 conn(0x7f7370007430 msgr2=0x7f73700098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:52:44.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:44.198+0000 7f738dd9b640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.122:3300/0 conn(0x7f7370007430 0x7f73700098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:52:44.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:44.198+0000 7f738dd9b640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.122:3300/0 conn(0x7f7370007430 0x7f73700098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T11:52:44.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:44.599+0000 7f738dd9b640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.122:3300/0 conn(0x7f7370007430 0x7f73700098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:44.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:44.599+0000 7f738dd9b640 10 monclient(hunting): get_auth_request con 0x7f7370007430 auth_method 0 2024-09-13T11:52:44.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:44.599+0000 7f738dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:44.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:44.599+0000 7f738dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:44.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:44.599+0000 7f738dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:52:44.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:44.599+0000 7f738dd9b640 1 -- 172.21.15.100:0/3886219220 >> v2:172.21.15.122:3300/0 conn(0x7f7370007430 msgr2=0x7f73700098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T11:52:44.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:44.599+0000 7f738dd9b640 1 -- 172.21.15.100:0/3886219220 >> v2:172.21.15.122:3300/0 conn(0x7f7370007430 msgr2=0x7f73700098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:52:44.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:44.599+0000 7f738dd9b640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.122:3300/0 conn(0x7f7370007430 0x7f73700098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:52:44.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:44.599+0000 7f738dd9b640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.122:3300/0 conn(0x7f7370007430 0x7f73700098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T11:52:45.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:45.400+0000 7f738dd9b640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.122:3300/0 conn(0x7f7370007430 0x7f73700098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:45.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:45.400+0000 7f738dd9b640 10 monclient(hunting): get_auth_request con 0x7f7370007430 auth_method 0 2024-09-13T11:52:45.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:45.400+0000 7f738dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:45.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:45.400+0000 7f738dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:45.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:45.400+0000 7f738dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:52:45.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:45.400+0000 7f738dd9b640 1 -- 172.21.15.100:0/3886219220 >> v2:172.21.15.122:3300/0 conn(0x7f7370007430 msgr2=0x7f73700098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T11:52:45.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:45.400+0000 7f738dd9b640 1 -- 172.21.15.100:0/3886219220 >> v2:172.21.15.122:3300/0 conn(0x7f7370007430 msgr2=0x7f73700098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:52:45.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:45.400+0000 7f738dd9b640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.122:3300/0 conn(0x7f7370007430 0x7f73700098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:52:45.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:45.400+0000 7f738dd9b640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.122:3300/0 conn(0x7f7370007430 0x7f73700098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T11:52:46.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 1 -- 172.21.15.100:0/3886219220 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4188658348 0 0) 0x7f739006fc60 con 0x7f73700036e0 2024-09-13T11:52:46.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 10 monclient(hunting): _negotiate 2024-09-13T11:52:46.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:46.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:46.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 10 monclient(hunting): my global_id is 4098 2024-09-13T11:52:46.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 1 -- 172.21.15.100:0/3886219220 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7378003520 con 0x7f73700036e0 2024-09-13T11:52:46.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 1 -- 172.21.15.100:0/3886219220 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2251039531 0 0) 0x7f7378003520 con 0x7f73700036e0 2024-09-13T11:52:46.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 1 -- 172.21.15.100:0/3886219220 >> v2:172.21.15.122:3300/0 conn(0x7f7370007430 msgr2=0x7f73700098b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:52:46.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 1 --2- 172.21.15.100:0/3886219220 >> v2:172.21.15.122:3300/0 conn(0x7f7370007430 0x7f73700098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 10 monclient: _finish_hunting 0 2024-09-13T11:52:46.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 1 monclient: found mon.noname-a-legacy 2024-09-13T11:52:46.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:52:46.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.100:6789/0 2024-09-13T11:52:46.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 1 -- 172.21.15.100:0/3886219220 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f737000afe0 con 0x7f73700036e0 2024-09-13T11:52:46.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 10 monclient: _finish_auth 0 2024-09-13T11:52:46.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 10 monclient: _check_auth_tickets 2024-09-13T11:52:46.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:52:46.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 1 -- 172.21.15.100:0/3886219220 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 626700363 0 0) 0x7f738000f040 con 0x7f73700036e0 2024-09-13T11:52:46.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f739495d640 5 monclient: authenticate success, global_id 4098 2024-09-13T11:52:46.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f739495d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:52:46.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:46.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-13T11:52:46.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 10 monclient: dump: 2024-09-13T11:52:46.186 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:52:46.186 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:46.186 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.186 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.186 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:46.186 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:46.186 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:46.186 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:46.186 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:46.186 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:52:46.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] but i'm connected to v1:172.21.15.100:6789/0, reconnecting 2024-09-13T11:52:46.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 10 monclient: _reopen_session rank -1 2024-09-13T11:52:46.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 1 -- 172.21.15.100:0/3886219220 >> v1:172.21.15.100:6789/0 conn(0x7f73700036e0 legacy=0x7f7370005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:46.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738ed9d640 1 -- 172.21.15.100:0/3886219220 reap_dead start 2024-09-13T11:52:46.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:52:46.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7378003a50 0x7f7378003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:46.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 10 monclient(hunting): picked mon.c con 0x7f7378003a50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:52:46.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7378004800 0x7f7378004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:46.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 10 monclient(hunting): picked mon.b con 0x7f7378004800 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:52:46.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73780053c0 0x7f737800b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:46.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 10 monclient(hunting): picked mon.a con 0x7f73780053c0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:52:46.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:46.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 1 -- 172.21.15.100:0/3886219220 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f73900a1e50 con 0x7f73780053c0 2024-09-13T11:52:46.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:46.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 1 -- 172.21.15.100:0/3886219220 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f737800c2c0 con 0x7f7378004800 2024-09-13T11:52:46.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:46.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 1 -- 172.21.15.100:0/3886219220 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f737800c4c0 con 0x7f7378003a50 2024-09-13T11:52:46.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 10 monclient(hunting): _renew_subs 2024-09-13T11:52:46.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738d59a640 10 monclient(hunting): _finish_auth 0 2024-09-13T11:52:46.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f739495d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-13T11:52:46.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738e59c640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7378003a50 0x7f7378003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:46.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738e59c640 10 monclient(hunting): get_auth_request con 0x7f7378003a50 auth_method 0 2024-09-13T11:52:46.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:46.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738e59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:46.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:52:46.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738ed9d640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73780053c0 0x7f737800b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:46.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738ed9d640 10 monclient(hunting): get_auth_request con 0x7f73780053c0 auth_method 0 2024-09-13T11:52:46.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:46.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:46.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:52:46.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:46.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:46.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738dd9b640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7378004800 0x7f7378004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:46.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738e59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:52:46.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738dd9b640 10 monclient(hunting): get_auth_request con 0x7f7378004800 auth_method 0 2024-09-13T11:52:46.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:46.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:46.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:52:46.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:46.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:46.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.174+0000 7f738ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:52:46.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.175+0000 7f738dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:46.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.175+0000 7f738dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:46.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.175+0000 7f738dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:52:46.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.175+0000 7f738e59c640 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2024-09-13T11:52:46.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.175+0000 7f738e59c640 1 -- 172.21.15.100:0/3886219220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7378004800 msgr2=0x7f7378004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:52:46.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.175+0000 7f738e59c640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7378004800 0x7f7378004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.175+0000 7f738e59c640 1 -- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73780053c0 msgr2=0x7f737800b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:52:46.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.175+0000 7f738e59c640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73780053c0 0x7f737800b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.175+0000 7f738e59c640 10 monclient: _finish_hunting 0 2024-09-13T11:52:46.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.175+0000 7f738e59c640 1 monclient: found mon.c 2024-09-13T11:52:46.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.175+0000 7f738e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:52:46.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.175+0000 7f738e59c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:52:46.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.175+0000 7f738e59c640 1 -- 172.21.15.100:0/3886219220 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7378003520 con 0x7f7378003a50 2024-09-13T11:52:46.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.175+0000 7f738e59c640 10 monclient: _finish_auth 0 2024-09-13T11:52:46.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.175+0000 7f738e59c640 10 monclient: _check_auth_tickets 2024-09-13T11:52:46.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.175+0000 7f738e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:52:46.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.175+0000 7f738e59c640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7378003a50 0x7f7378003e50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f738000cfc0 tx=0x7f7378005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcd8d0420941ce92 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:52:46.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.176+0000 7f738d59a640 1 -- 172.21.15.100:0/3886219220 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7380031070 con 0x7f7378003a50 2024-09-13T11:52:46.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.176+0000 7f738d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:46.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.176+0000 7f738d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:52:46.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.176+0000 7f738d59a640 10 monclient: dump: 2024-09-13T11:52:46.191 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:52:46.191 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:46.191 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.192 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.192 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:46.192 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:46.192 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:46.192 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:46.192 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:46.192 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:52:46.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.176+0000 7f738d59a640 1 -- 172.21.15.100:0/3886219220 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7380035620 con 0x7f7378003a50 2024-09-13T11:52:46.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.176+0000 7f738d59a640 10 monclient: handle_config config(0 keys) 2024-09-13T11:52:46.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.176+0000 7f738d59a640 1 -- 172.21.15.100:0/3886219220 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738000cc10 con 0x7f7378003a50 2024-09-13T11:52:46.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.176+0000 7f739495d640 10 monclient: get_monmap_and_config success 2024-09-13T11:52:46.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.176+0000 7f739495d640 4 set_mon_vals no callback set 2024-09-13T11:52:46.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.176+0000 7f738d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:46.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.176+0000 7f738d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:52:46.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.176+0000 7f738d59a640 10 monclient: dump: 2024-09-13T11:52:46.193 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:52:46.193 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:46.193 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.193 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.193 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:46.193 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:46.193 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:46.193 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:46.193 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:46.193 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:52:46.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.176+0000 7f739495d640 10 monclient: shutdown 2024-09-13T11:52:46.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.176+0000 7f739495d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:52:46.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.176+0000 7f739495d640 1 -- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7378003a50 msgr2=0x7f7378003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:46.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.176+0000 7f739495d640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7378003a50 0x7f7378003e50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f738000cfc0 tx=0x7f7378005830 comp rx=0 tx=0).stop 2024-09-13T11:52:46.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.176+0000 7f739495d640 1 -- 172.21.15.100:0/3886219220 shutdown_connections 2024-09-13T11:52:46.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.176+0000 7f739495d640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73780053c0 0x7f737800b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.176+0000 7f739495d640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7378004800 0x7f7378004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.176+0000 7f739495d640 1 --2- 172.21.15.100:0/3886219220 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7378003a50 0x7f7378003e50 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.176+0000 7f739495d640 1 -- 172.21.15.100:0/3886219220 >> 172.21.15.100:0/3886219220 conn(0x7f73900a6970 msgr2=0x7f73900a6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:52:46.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.177+0000 7f739495d640 1 -- 172.21.15.100:0/3886219220 shutdown_connections 2024-09-13T11:52:46.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.177+0000 7f739495d640 1 -- 172.21.15.100:0/3886219220 wait complete. 2024-09-13T11:52:46.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.178+0000 7f739495d640 10 monclient: build_initial_monmap 2024-09-13T11:52:46.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.178+0000 7f739495d640 10 monclient: monmap: 2024-09-13T11:52:46.194 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:52:46.195 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:52:46.195 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:52:46.195 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:52:46.195 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:52:46.195 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:46.195 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:52:46.195 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:52:46.195 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:52:46.195 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:52:46.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.178+0000 7f739495d640 1 Processor -- start 2024-09-13T11:52:46.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.178+0000 7f739495d640 1 -- start start 2024-09-13T11:52:46.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.178+0000 7f739495d640 10 monclient: init 2024-09-13T11:52:46.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.178+0000 7f739495d640 10 monclient: _reopen_session rank -1 2024-09-13T11:52:46.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.178+0000 7f739495d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:52:46.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.178+0000 7f739495d640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7378003a50 0x7f73901cc150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:46.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.178+0000 7f739495d640 10 monclient(hunting): picked mon.noname-a con 0x7f7378003a50 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:52:46.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.178+0000 7f739495d640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7378004800 0x7f7390135c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:46.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.178+0000 7f739495d640 10 monclient(hunting): picked mon.noname-c con 0x7f7378004800 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:52:46.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.178+0000 7f739495d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73780053c0 0x7f7390136170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:46.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.178+0000 7f739495d640 10 monclient(hunting): picked mon.noname-b con 0x7f73780053c0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:52:46.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.178+0000 7f739495d640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:46.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.178+0000 7f739495d640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f73900a29b0 con 0x7f7378003a50 2024-09-13T11:52:46.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.178+0000 7f739495d640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:46.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.178+0000 7f739495d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f73900a2830 con 0x7f73780053c0 2024-09-13T11:52:46.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.178+0000 7f739495d640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:46.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.178+0000 7f739495d640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f73900a2b30 con 0x7f7378004800 2024-09-13T11:52:46.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.178+0000 7f739495d640 10 monclient(hunting): _renew_subs 2024-09-13T11:52:46.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.178+0000 7f739495d640 10 monclient(hunting): authenticate will time out at 746.771994s 2024-09-13T11:52:46.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738dd9b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7378004800 0x7f7390135c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:46.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738e59c640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7378003a50 0x7f73901cc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:46.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738dd9b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7378004800 0x7f7390135c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:51248/0 (socket says 172.21.15.100:51248) 2024-09-13T11:52:46.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738dd9b640 1 -- 172.21.15.100:0/2917858553 learned_addr learned my addr 172.21.15.100:0/2917858553 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:52:46.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738dd9b640 10 monclient(hunting): get_auth_request con 0x7f7378004800 auth_method 0 2024-09-13T11:52:46.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:46.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:46.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:46.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738e59c640 10 monclient(hunting): get_auth_request con 0x7f7378003a50 auth_method 0 2024-09-13T11:52:46.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:46.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738e59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:46.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:46.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:46.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:46.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:46.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:46.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:46.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:46.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738ed9d640 1 --2- 172.21.15.100:0/2917858553 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73780053c0 0x7f7390136170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:46.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738dd9b640 10 monclient(hunting): handle_auth_done global_id 4100 payload 824 2024-09-13T11:52:46.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738dd9b640 1 -- 172.21.15.100:0/2917858553 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73780053c0 msgr2=0x7f7390136170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:46.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738dd9b640 1 --2- 172.21.15.100:0/2917858553 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73780053c0 0x7f7390136170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738dd9b640 1 -- 172.21.15.100:0/2917858553 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7378003a50 msgr2=0x7f73901cc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:46.199 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738dd9b640 1 --2- 172.21.15.100:0/2917858553 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7378003a50 0x7f73901cc150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.199 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738dd9b640 10 monclient: _finish_hunting 0 2024-09-13T11:52:46.199 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738dd9b640 1 monclient: found mon.noname-c 2024-09-13T11:52:46.199 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:52:46.199 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:52:46.199 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738dd9b640 1 -- 172.21.15.100:0/2917858553 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7390122320 con 0x7f7378004800 2024-09-13T11:52:46.199 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738ed9d640 10 monclient: get_auth_request con 0x7f73780053c0 auth_method 0 2024-09-13T11:52:46.199 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738ed9d640 1 --2- 172.21.15.100:0/2917858553 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73780053c0 0x7f7390136170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:52:46.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738dd9b640 1 --2- 172.21.15.100:0/2917858553 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7378004800 0x7f7390135c30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7384016470 tx=0x7f7384016930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:52:46.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f737effd640 1 -- 172.21.15.100:0/2917858553 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7384007260 con 0x7f7378004800 2024-09-13T11:52:46.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f737effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:46.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f737effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:52:46.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f737effd640 10 monclient: dump: 2024-09-13T11:52:46.200 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:52:46.200 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:46.200 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.200 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.200 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:46.200 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:46.200 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:46.201 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:46.201 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:46.201 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:52:46.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f737effd640 10 monclient: _finish_auth 0 2024-09-13T11:52:46.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f737effd640 10 monclient: _check_auth_tickets 2024-09-13T11:52:46.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f737effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:52:46.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f737effd640 1 -- 172.21.15.100:0/2917858553 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f738401b030 con 0x7f7378004800 2024-09-13T11:52:46.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f739495d640 5 monclient: authenticate success, global_id 4100 2024-09-13T11:52:46.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f739495d640 10 monclient: _renew_subs 2024-09-13T11:52:46.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f739495d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:52:46.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f739495d640 1 -- 172.21.15.100:0/2917858553 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7390122600 con 0x7f7378004800 2024-09-13T11:52:46.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f737effd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:52:46.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f737effd640 1 -- 172.21.15.100:0/2917858553 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7384007620 con 0x7f7378004800 2024-09-13T11:52:46.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f737effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:46.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f737effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:52:46.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f737effd640 10 monclient: dump: 2024-09-13T11:52:46.202 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:52:46.202 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:46.202 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.202 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.202 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:46.202 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:46.202 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:46.202 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:46.202 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:46.203 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:52:46.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f739495d640 10 monclient: _renew_subs 2024-09-13T11:52:46.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f738cd99640 4 set_mon_vals no callback set 2024-09-13T11:52:46.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f739495d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:52:46.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.179+0000 7f739495d640 1 -- 172.21.15.100:0/2917858553 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f73900a0ac0 con 0x7f7378004800 2024-09-13T11:52:46.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.180+0000 7f737effd640 1 -- 172.21.15.100:0/2917858553 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7384013070 con 0x7f7378004800 2024-09-13T11:52:46.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.180+0000 7f737effd640 1 -- 172.21.15.100:0/2917858553 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7384020760 con 0x7f7378004800 2024-09-13T11:52:46.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.181+0000 7f739495d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:46.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.181+0000 7f739495d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:46.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.181+0000 7f739495d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:52:46.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.181+0000 7f739495d640 1 -- 172.21.15.100:0/2917858553 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f739006f210 con 0x7f7378004800 2024-09-13T11:52:46.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.183+0000 7f737effd640 1 -- 172.21.15.100:0/2917858553 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f7384006cf0 con 0x7f7378004800 2024-09-13T11:52:46.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.183+0000 7f737effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:46.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.183+0000 7f737effd640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:52:46.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.246+0000 7f739495d640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-13T11:52:46.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.246+0000 7f739495d640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-13T11:52:46.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.246+0000 7f739495d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:52:46.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.246+0000 7f739495d640 1 -- 172.21.15.100:0/2917858553 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f7390121f60 con 0x7f7378004800 2024-09-13T11:52:46.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.251+0000 7f737effd640 1 -- 172.21.15.100:0/2917858553 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f7384017c80 con 0x7f7378004800 2024-09-13T11:52:46.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.252+0000 7f737effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-13T11:52:46.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.252+0000 7f737effd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-13T11:52:46.253 INFO:teuthology.orchestra.run.smithi100.stderr:adjusted tunables profile to default 2024-09-13T11:52:46.254 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.252+0000 7f739495d640 10 monclient: shutdown 2024-09-13T11:52:46.254 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.252+0000 7f739495d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:52:46.254 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.252+0000 7f739495d640 1 -- 172.21.15.100:0/2917858553 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7378004800 msgr2=0x7f7390135c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:46.254 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.252+0000 7f739495d640 1 --2- 172.21.15.100:0/2917858553 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7378004800 0x7f7390135c30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7384016470 tx=0x7f7384016930 comp rx=0 tx=0).stop 2024-09-13T11:52:46.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.252+0000 7f739495d640 1 -- 172.21.15.100:0/2917858553 shutdown_connections 2024-09-13T11:52:46.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.252+0000 7f739495d640 1 --2- 172.21.15.100:0/2917858553 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73780053c0 0x7f7390136170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.252+0000 7f739495d640 1 --2- 172.21.15.100:0/2917858553 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7378004800 0x7f7390135c30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.252+0000 7f739495d640 1 --2- 172.21.15.100:0/2917858553 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7378003a50 0x7f73901cc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.252+0000 7f739495d640 1 -- 172.21.15.100:0/2917858553 >> 172.21.15.100:0/2917858553 conn(0x7f73900a6970 msgr2=0x7f7390138d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:52:46.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.253+0000 7f739495d640 1 -- 172.21.15.100:0/2917858553 shutdown_connections 2024-09-13T11:52:46.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:46.253+0000 7f739495d640 1 -- 172.21.15.100:0/2917858553 wait complete. 2024-09-13T11:52:46.266 INFO:tasks.ceph:check_enable_crimson: False 2024-09-13T11:52:46.266 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-13T11:52:46.266 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T11:52:46.266 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-13T11:52:46.291 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T11:52:46.291 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-13T11:52:46.356 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T11:52:46.356 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-13T11:52:46.420 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T11:52:46.421 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-13T11:52:46.451 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T11:52:46.452 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-13T11:52:46.522 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T11:52:46.522 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-13T11:52:46.592 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new ac784459-dbff-42ed-941e-70c741392dcf 0 2024-09-13T11:52:46.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.745+0000 7f60d9095640 10 monclient: get_monmap_and_config 2024-09-13T11:52:46.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.745+0000 7f60d9095640 10 monclient: build_initial_monmap 2024-09-13T11:52:46.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.745+0000 7f60d9095640 10 monclient: monmap: 2024-09-13T11:52:46.747 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 0 2024-09-13T11:52:46.747 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:46.747 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:46.746465+0000 2024-09-13T11:52:46.748 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:46.746465+0000 2024-09-13T11:52:46.748 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 0 (unknown) 2024-09-13T11:52:46.748 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:46.748 INFO:teuthology.orchestra.run.smithi122.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:52:46.748 INFO:teuthology.orchestra.run.smithi122.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:52:46.748 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:52:46.748 INFO:teuthology.orchestra.run.smithi122.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:52:46.748 INFO:teuthology.orchestra.run.smithi122.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:52:46.748 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:46.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.745+0000 7f60d9095640 1 Processor -- start 2024-09-13T11:52:46.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.745+0000 7f60d9095640 1 -- start start 2024-09-13T11:52:46.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.745+0000 7f60d9095640 10 monclient: init 2024-09-13T11:52:46.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d9095640 10 monclient: _reopen_session rank -1 2024-09-13T11:52:46.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d9095640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-13T11:52:46.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d9095640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f60d406ea50 0x7f60d40a2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:46.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d9095640 10 monclient(hunting): picked mon.noname-c con 0x7f60d406ea50 addr v2:172.21.15.122:3300/0 2024-09-13T11:52:46.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d9095640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f60d406ee30 addr v1:172.21.15.122:6789/0 2024-09-13T11:52:46.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d9095640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d413caf0 0x7f60d413eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:46.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d9095640 10 monclient(hunting): picked mon.noname-b con 0x7f60d413caf0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:52:46.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d9095640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60d40a6440 con 0x7f60d406ee30 2024-09-13T11:52:46.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d9095640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:46.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d9095640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f60d40a1e50 con 0x7f60d406ea50 2024-09-13T11:52:46.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d9095640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:46.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d9095640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f60d40a6680 con 0x7f60d413caf0 2024-09-13T11:52:46.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d9095640 10 monclient(hunting): _renew_subs 2024-09-13T11:52:46.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d9095640 10 monclient(hunting): authenticate will time out at 747.059670s 2024-09-13T11:52:46.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d2d9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f60d406ea50 0x7f60d40a2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:46.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d2d9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f60d406ea50 0x7f60d40a2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:50042/0 (socket says 172.21.15.122:50042) 2024-09-13T11:52:46.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d2d9d640 1 -- 172.21.15.122:0/2559712589 learned_addr learned my addr 172.21.15.122:0/2559712589 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-09-13T11:52:46.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d2d9d640 10 monclient(hunting): get_auth_request con 0x7f60d406ea50 auth_method 0 2024-09-13T11:52:46.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:46.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:46.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:46.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d259c640 1 --1- 172.21.15.122:0/2559712589 >> v1:172.21.15.122:6789/0 conn(0x7f60d406ee30 0x7f60d406f240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:56254/0 (socket says 172.21.15.122:56254) 2024-09-13T11:52:46.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:46.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:46.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.746+0000 7f60d359e640 1 --2- 172.21.15.122:0/2559712589 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d413caf0 0x7f60d413eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:46.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.747+0000 7f60d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:46.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.747+0000 7f60d359e640 10 monclient(hunting): get_auth_request con 0x7f60d413caf0 auth_method 0 2024-09-13T11:52:46.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.747+0000 7f60d359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:46.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.747+0000 7f60d359e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:46.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.747+0000 7f60d359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:46.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.747+0000 7f60d359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:46.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.747+0000 7f60d359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:46.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.747+0000 7f60d359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:46.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.747+0000 7f60d2d9d640 10 monclient(hunting): handle_auth_done global_id 4099 payload 274 2024-09-13T11:52:46.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.747+0000 7f60d2d9d640 1 -- 172.21.15.122:0/2559712589 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d413caf0 msgr2=0x7f60d413eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:52:46.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.747+0000 7f60d2d9d640 1 --2- 172.21.15.122:0/2559712589 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d413caf0 0x7f60d413eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.747+0000 7f60d2d9d640 1 -- 172.21.15.122:0/2559712589 >> v1:172.21.15.122:6789/0 conn(0x7f60d406ee30 legacy=0x7f60d406f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:46.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.747+0000 7f60d2d9d640 10 monclient: _finish_hunting 0 2024-09-13T11:52:46.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.747+0000 7f60d2d9d640 1 monclient: found mon.noname-c 2024-09-13T11:52:46.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.747+0000 7f60d2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:52:46.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.747+0000 7f60d2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.122:3300/0 2024-09-13T11:52:46.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.747+0000 7f60d2d9d640 1 -- 172.21.15.122:0/2559712589 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d406fc60 con 0x7f60d406ea50 2024-09-13T11:52:46.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d2d9d640 1 --2- 172.21.15.122:0/2559712589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d406ea50 0x7f60d40a2830 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f60c4009870 tx=0x7f60c402a130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1931b752b92ed27d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:52:46.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d1d9b640 1 -- 172.21.15.122:0/2559712589 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c4037070 con 0x7f60d406ea50 2024-09-13T11:52:46.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:46.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:52:46.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d1d9b640 10 monclient: dump: 2024-09-13T11:52:46.763 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-09-13T11:52:46.763 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:46.763 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.763 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.763 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:46.763 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:46.763 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:46.763 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:46.763 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:46.763 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:46.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d1d9b640 10 monclient: _finish_auth 0 2024-09-13T11:52:46.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d1d9b640 10 monclient: _check_auth_tickets 2024-09-13T11:52:46.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:52:46.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d1d9b640 1 -- 172.21.15.122:0/2559712589 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60c4032440 con 0x7f60d406ea50 2024-09-13T11:52:46.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d9095640 5 monclient: authenticate success, global_id 4099 2024-09-13T11:52:46.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d9095640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:52:46.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d1d9b640 10 monclient: handle_config config(0 keys) 2024-09-13T11:52:46.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d9095640 10 monclient: get_monmap_and_config success 2024-09-13T11:52:46.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d1d9b640 1 -- 172.21.15.122:0/2559712589 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c4032810 con 0x7f60d406ea50 2024-09-13T11:52:46.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:46.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d9095640 4 set_mon_vals no callback set 2024-09-13T11:52:46.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:52:46.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d1d9b640 10 monclient: dump: 2024-09-13T11:52:46.765 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-09-13T11:52:46.765 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:46.765 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.765 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.765 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:46.765 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:46.765 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:46.765 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:46.765 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:46.766 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:46.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d9095640 10 monclient: shutdown 2024-09-13T11:52:46.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d9095640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:52:46.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d9095640 1 -- 172.21.15.122:0/2559712589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d406ea50 msgr2=0x7f60d40a2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:46.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d9095640 1 --2- 172.21.15.122:0/2559712589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d406ea50 0x7f60d40a2830 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f60c4009870 tx=0x7f60c402a130 comp rx=0 tx=0).stop 2024-09-13T11:52:46.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d9095640 1 -- 172.21.15.122:0/2559712589 shutdown_connections 2024-09-13T11:52:46.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d9095640 1 --2- 172.21.15.122:0/2559712589 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d413caf0 0x7f60d413eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.748+0000 7f60d9095640 1 --2- 172.21.15.122:0/2559712589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d406ea50 0x7f60d40a2830 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.749+0000 7f60d9095640 1 -- 172.21.15.122:0/2559712589 >> 172.21.15.122:0/2559712589 conn(0x7f60d40a6970 msgr2=0x7f60d40a6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:52:46.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.749+0000 7f60d9095640 1 -- 172.21.15.122:0/2559712589 shutdown_connections 2024-09-13T11:52:46.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.749+0000 7f60d9095640 1 -- 172.21.15.122:0/2559712589 wait complete. 2024-09-13T11:52:46.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.750+0000 7f60d9095640 10 monclient: build_initial_monmap 2024-09-13T11:52:46.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.750+0000 7f60d9095640 10 monclient: monmap: 2024-09-13T11:52:46.766 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 0 2024-09-13T11:52:46.767 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:52:46.767 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 0.000000 2024-09-13T11:52:46.767 INFO:teuthology.orchestra.run.smithi122.stderr:created 0.000000 2024-09-13T11:52:46.767 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 0 (unknown) 2024-09-13T11:52:46.767 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:46.767 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:52:46.767 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:52:46.767 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:52:46.767 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:46.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.750+0000 7f60d9095640 1 Processor -- start 2024-09-13T11:52:46.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.750+0000 7f60d9095640 1 -- start start 2024-09-13T11:52:46.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.750+0000 7f60d9095640 10 monclient: init 2024-09-13T11:52:46.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d9095640 10 monclient: _reopen_session rank -1 2024-09-13T11:52:46.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d9095640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:52:46.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d9095640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60d406ea50 0x7f60d4120900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:46.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d9095640 10 monclient(hunting): picked mon.noname-a con 0x7f60d406ea50 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:52:46.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d9095640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d406ee30 0x7f60d4137770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:46.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d9095640 10 monclient(hunting): picked mon.noname-c con 0x7f60d406ee30 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:52:46.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d9095640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d413caf0 0x7f60d409eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:46.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d9095640 10 monclient(hunting): picked mon.noname-b con 0x7f60d413caf0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:52:46.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d9095640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:46.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d9095640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f60d413f5a0 con 0x7f60d406ea50 2024-09-13T11:52:46.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d9095640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:46.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d9095640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f60d413f420 con 0x7f60d413caf0 2024-09-13T11:52:46.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d9095640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:46.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d9095640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f60d413f720 con 0x7f60d406ee30 2024-09-13T11:52:46.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d9095640 10 monclient(hunting): _renew_subs 2024-09-13T11:52:46.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d9095640 10 monclient(hunting): authenticate will time out at 747.064371s 2024-09-13T11:52:46.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d359e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d413caf0 0x7f60d409eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:46.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d359e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d413caf0 0x7f60d409eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:50050/0 (socket says 172.21.15.122:50050) 2024-09-13T11:52:46.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d359e640 1 -- 172.21.15.122:0/9439764 learned_addr learned my addr 172.21.15.122:0/9439764 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-09-13T11:52:46.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d359e640 10 monclient(hunting): get_auth_request con 0x7f60d413caf0 auth_method 0 2024-09-13T11:52:46.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:46.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d359e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:46.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:46.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d259c640 1 --2- 172.21.15.122:0/9439764 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d406ee30 0x7f60d4137770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:46.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d259c640 10 monclient(hunting): get_auth_request con 0x7f60d406ee30 auth_method 0 2024-09-13T11:52:46.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:46.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d259c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:46.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:46.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d2d9d640 1 --2- 172.21.15.122:0/9439764 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60d406ea50 0x7f60d4120900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:46.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:46.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:46.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:46.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d2d9d640 10 monclient(hunting): get_auth_request con 0x7f60d406ea50 auth_method 0 2024-09-13T11:52:46.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:46.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:46.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:46.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:46.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:46.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:46.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:46.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:46.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:46.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d359e640 10 monclient(hunting): handle_auth_done global_id 4105 payload 824 2024-09-13T11:52:46.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d359e640 1 -- 172.21.15.122:0/9439764 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d406ee30 msgr2=0x7f60d4137770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:46.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d359e640 1 --2- 172.21.15.122:0/9439764 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d406ee30 0x7f60d4137770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d359e640 1 -- 172.21.15.122:0/9439764 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60d406ea50 msgr2=0x7f60d4120900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:46.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d359e640 1 --2- 172.21.15.122:0/9439764 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60d406ea50 0x7f60d4120900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d359e640 10 monclient: _finish_hunting 0 2024-09-13T11:52:46.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d359e640 1 monclient: found mon.noname-b 2024-09-13T11:52:46.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:52:46.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d359e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.122:3300/0 2024-09-13T11:52:46.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.751+0000 7f60d359e640 1 -- 172.21.15.122:0/9439764 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d40a0ab0 con 0x7f60d413caf0 2024-09-13T11:52:46.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d359e640 1 --2- 172.21.15.122:0/9439764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d413caf0 0x7f60d409eb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f60cc009d20 tx=0x7f60cc00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:52:46.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d359e640 1 -- 172.21.15.122:0/9439764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d413caf0 msgr2=0x7f60d409eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-13T11:52:46.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d359e640 1 -- 172.21.15.122:0/9439764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d413caf0 msgr2=0x7f60d409eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T11:52:46.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d359e640 1 --2- 172.21.15.122:0/9439764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d413caf0 0x7f60d409eb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f60cc009d20 tx=0x7f60cc00c490 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:52:46.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d359e640 1 --2- 172.21.15.122:0/9439764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d413caf0 0x7f60d409eb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f60cc009d20 tx=0x7f60cc00c490 comp rx=0 tx=0).stop 2024-09-13T11:52:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60bb7fe640 10 monclient: ms_handle_reset current mon [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:52:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60bb7fe640 10 monclient: _reopen_session rank -1 2024-09-13T11:52:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60bb7fe640 1 -- 172.21.15.122:0/9439764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d413caf0 msgr2=0x7f60d409eb70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:52:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60bb7fe640 1 --2- 172.21.15.122:0/9439764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d413caf0 0x7f60d409eb70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60bb7fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:52:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60bb7fe640 1 --2- 172.21.15.122:0/9439764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60c00034a0 0x7f60c00058f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60bb7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f60c00034a0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:52:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60bb7fe640 1 --2- 172.21.15.122:0/9439764 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d406ee30 0x7f60c0008460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60bb7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f60d406ee30 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:52:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60bb7fe640 1 --2- 172.21.15.122:0/9439764 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60d406ea50 0x7f60c000af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60bb7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f60d406ea50 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:52:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60bb7fe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60bb7fe640 1 -- 172.21.15.122:0/9439764 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f60c000b940 con 0x7f60d406ea50 2024-09-13T11:52:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60bb7fe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60bb7fe640 1 -- 172.21.15.122:0/9439764 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f60c000bac0 con 0x7f60c00034a0 2024-09-13T11:52:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60bb7fe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60bb7fe640 1 -- 172.21.15.122:0/9439764 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f60c000bcc0 con 0x7f60d406ee30 2024-09-13T11:52:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60bb7fe640 10 monclient(hunting): _renew_subs 2024-09-13T11:52:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d2d9d640 1 --2- 172.21.15.122:0/9439764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60c00034a0 0x7f60c00058f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d2d9d640 10 monclient(hunting): get_auth_request con 0x7f60c00034a0 auth_method 0 2024-09-13T11:52:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d259c640 1 --2- 172.21.15.122:0/9439764 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d406ee30 0x7f60c0008460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d359e640 1 --2- 172.21.15.122:0/9439764 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60d406ea50 0x7f60c000af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d2d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:52:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d359e640 10 monclient(hunting): get_auth_request con 0x7f60d406ea50 auth_method 0 2024-09-13T11:52:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:46.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d359e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:46.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d359e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:52:46.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d259c640 10 monclient(hunting): get_auth_request con 0x7f60d406ee30 auth_method 0 2024-09-13T11:52:46.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:46.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d259c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:46.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:52:46.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:46.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:46.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:52:46.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:46.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:46.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d359e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:52:46.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:46.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:46.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:52:46.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d2d9d640 10 monclient(hunting): handle_auth_done global_id 4105 payload 293 2024-09-13T11:52:46.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d2d9d640 1 -- 172.21.15.122:0/9439764 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d406ee30 msgr2=0x7f60c0008460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:46.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d2d9d640 1 --2- 172.21.15.122:0/9439764 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d406ee30 0x7f60c0008460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d2d9d640 1 -- 172.21.15.122:0/9439764 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60d406ea50 msgr2=0x7f60c000af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:46.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d2d9d640 1 --2- 172.21.15.122:0/9439764 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60d406ea50 0x7f60c000af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d2d9d640 10 monclient: _finish_hunting 0 2024-09-13T11:52:46.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d2d9d640 1 monclient: found mon.noname-b 2024-09-13T11:52:46.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:52:46.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.122:3300/0 2024-09-13T11:52:46.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d2d9d640 1 -- 172.21.15.122:0/9439764 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60c000bf80 con 0x7f60c00034a0 2024-09-13T11:52:46.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d359e640 1 --2- 172.21.15.122:0/9439764 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60d406ea50 0x7f60c000af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:52:46.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.752+0000 7f60d2d9d640 1 --2- 172.21.15.122:0/9439764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60c00034a0 0x7f60c00058f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f60c4031000 tx=0x7f60cc00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:52:46.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.753+0000 7f60bb7fe640 1 -- 172.21.15.122:0/9439764 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c4037040 con 0x7f60c00034a0 2024-09-13T11:52:46.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.753+0000 7f60bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:46.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.753+0000 7f60bb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:52:46.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.753+0000 7f60bb7fe640 10 monclient: dump: 2024-09-13T11:52:46.777 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-09-13T11:52:46.777 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:46.777 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.777 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.777 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:46.778 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:46.778 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:46.778 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:46.778 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:46.778 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:46.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.753+0000 7f60bb7fe640 10 monclient: _finish_auth 0 2024-09-13T11:52:46.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.753+0000 7f60bb7fe640 10 monclient: _check_auth_tickets 2024-09-13T11:52:46.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.753+0000 7f60bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:52:46.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.753+0000 7f60bb7fe640 1 -- 172.21.15.122:0/9439764 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60c4032c00 con 0x7f60c00034a0 2024-09-13T11:52:46.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.753+0000 7f60d9095640 5 monclient: authenticate success, global_id 4105 2024-09-13T11:52:46.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.753+0000 7f60d9095640 10 monclient: _renew_subs 2024-09-13T11:52:46.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.753+0000 7f60d9095640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:52:46.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.753+0000 7f60d9095640 1 -- 172.21.15.122:0/9439764 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60d41cd460 con 0x7f60c00034a0 2024-09-13T11:52:46.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.753+0000 7f60bb7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:52:46.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.753+0000 7f60bb7fe640 1 -- 172.21.15.122:0/9439764 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c403c440 con 0x7f60c00034a0 2024-09-13T11:52:46.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.753+0000 7f60bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:46.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.753+0000 7f60bb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:52:46.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.753+0000 7f60bb7fe640 10 monclient: dump: 2024-09-13T11:52:46.779 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-09-13T11:52:46.779 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:46.779 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.779 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.780 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:46.780 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:46.780 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:46.780 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:46.780 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:46.780 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:46.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.753+0000 7f60d9095640 10 monclient: _renew_subs 2024-09-13T11:52:46.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.753+0000 7f60d159a640 4 set_mon_vals no callback set 2024-09-13T11:52:46.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.753+0000 7f60d9095640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:52:46.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.753+0000 7f60d9095640 1 -- 172.21.15.122:0/9439764 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60d413ec10 con 0x7f60c00034a0 2024-09-13T11:52:46.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.753+0000 7f60bb7fe640 1 -- 172.21.15.122:0/9439764 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f60c403a040 con 0x7f60c00034a0 2024-09-13T11:52:46.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.753+0000 7f60bb7fe640 1 -- 172.21.15.122:0/9439764 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f60c4048a40 con 0x7f60c00034a0 2024-09-13T11:52:46.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.754+0000 7f60d9095640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:46.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.754+0000 7f60d9095640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:46.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.754+0000 7f60d9095640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:52:46.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.754+0000 7f60d9095640 1 -- 172.21.15.122:0/9439764 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60d40a2830 con 0x7f60c00034a0 2024-09-13T11:52:46.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.756+0000 7f60bb7fe640 1 -- 172.21.15.122:0/9439764 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f60d41cd460 con 0x7f60c00034a0 2024-09-13T11:52:46.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.756+0000 7f60bb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:46.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.756+0000 7f60bb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:52:46.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.827+0000 7f60d9095640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ac784459-dbff-42ed-941e-70c741392dcf", "id": 0}] 2024-09-13T11:52:46.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.828+0000 7f60d9095640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ac784459-dbff-42ed-941e-70c741392dcf", "id": 0}] 2024-09-13T11:52:46.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.828+0000 7f60d9095640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:52:46.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.828+0000 7f60d9095640 1 -- 172.21.15.122:0/9439764 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ac784459-dbff-42ed-941e-70c741392dcf", "id": 0} v 0) -- 0x7f60d4069100 con 0x7f60c00034a0 2024-09-13T11:52:46.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.833+0000 7f60bb7fe640 1 -- 172.21.15.122:0/9439764 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ac784459-dbff-42ed-941e-70c741392dcf", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f60c4048ce0 con 0x7f60c00034a0 2024-09-13T11:52:46.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.833+0000 7f60bb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ac784459-dbff-42ed-941e-70c741392dcf", "id": 0}] 2024-09-13T11:52:46.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.833+0000 7f60bb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:52:46.835 INFO:teuthology.orchestra.run.smithi122.stdout:0 2024-09-13T11:52:46.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.834+0000 7f60d9095640 10 monclient: shutdown 2024-09-13T11:52:46.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.834+0000 7f60d9095640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:52:46.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.834+0000 7f60d9095640 1 -- 172.21.15.122:0/9439764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60c00034a0 msgr2=0x7f60c00058f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:46.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.834+0000 7f60d9095640 1 --2- 172.21.15.122:0/9439764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60c00034a0 0x7f60c00058f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f60c4031000 tx=0x7f60cc00b040 comp rx=0 tx=0).stop 2024-09-13T11:52:46.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.834+0000 7f60d9095640 1 -- 172.21.15.122:0/9439764 shutdown_connections 2024-09-13T11:52:46.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.834+0000 7f60d9095640 1 --2- 172.21.15.122:0/9439764 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60d406ea50 0x7f60c000af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.834+0000 7f60d9095640 1 --2- 172.21.15.122:0/9439764 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60d406ee30 0x7f60c0008460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.834+0000 7f60d9095640 1 --2- 172.21.15.122:0/9439764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60c00034a0 0x7f60c00058f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.834+0000 7f60d9095640 1 -- 172.21.15.122:0/9439764 >> 172.21.15.122:0/9439764 conn(0x7f60d40a6970 msgr2=0x7f60d413bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:52:46.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.834+0000 7f60d9095640 1 -- 172.21.15.122:0/9439764 shutdown_connections 2024-09-13T11:52:46.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.834+0000 7f60d9095640 1 -- 172.21.15.122:0/9439764 wait complete. 2024-09-13T11:52:46.847 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new 8d8d39e7-e8f1-499a-a01b-e7ba49844eae 1 2024-09-13T11:52:46.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.942+0000 7fdbb70bf640 10 monclient: get_monmap_and_config 2024-09-13T11:52:46.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.942+0000 7fdbb70bf640 10 monclient: build_initial_monmap 2024-09-13T11:52:46.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.942+0000 7fdbb70bf640 10 monclient: monmap: 2024-09-13T11:52:46.945 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 0 2024-09-13T11:52:46.945 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:46.945 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:46.944034+0000 2024-09-13T11:52:46.945 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:46.944034+0000 2024-09-13T11:52:46.945 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 0 (unknown) 2024-09-13T11:52:46.945 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:46.945 INFO:teuthology.orchestra.run.smithi122.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:52:46.945 INFO:teuthology.orchestra.run.smithi122.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:52:46.945 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:52:46.945 INFO:teuthology.orchestra.run.smithi122.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:52:46.945 INFO:teuthology.orchestra.run.smithi122.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:52:46.945 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:46.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.943+0000 7fdbb70bf640 1 Processor -- start 2024-09-13T11:52:46.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.943+0000 7fdbb70bf640 1 -- start start 2024-09-13T11:52:46.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.943+0000 7fdbb70bf640 10 monclient: init 2024-09-13T11:52:46.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.943+0000 7fdbb70bf640 10 monclient: _reopen_session rank -1 2024-09-13T11:52:46.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.943+0000 7fdbb70bf640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-13T11:52:46.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.943+0000 7fdbb70bf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdbb006ea50 addr v1:172.21.15.100:6789/0 2024-09-13T11:52:46.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.943+0000 7fdbb70bf640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdbb006f540 0x7fdbb013caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:46.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.943+0000 7fdbb70bf640 10 monclient(hunting): picked mon.noname-b con 0x7fdbb006f540 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:52:46.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.943+0000 7fdbb70bf640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fdbb013d030 0x7fdbb013f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:46.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.943+0000 7fdbb70bf640 10 monclient(hunting): picked mon.noname-c con 0x7fdbb013d030 addr v2:172.21.15.122:3300/0 2024-09-13T11:52:46.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.943+0000 7fdbb70bf640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbb00a1e20 con 0x7fdbb006ea50 2024-09-13T11:52:46.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.943+0000 7fdbb70bf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:46.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.943+0000 7fdbb70bf640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7fdbb00a0ec0 con 0x7fdbb013d030 2024-09-13T11:52:46.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.943+0000 7fdbb70bf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:46.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.943+0000 7fdbb70bf640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fdbb00a1b40 con 0x7fdbb006f540 2024-09-13T11:52:46.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.943+0000 7fdbb70bf640 10 monclient(hunting): _renew_subs 2024-09-13T11:52:46.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.943+0000 7fdbb70bf640 10 monclient(hunting): authenticate will time out at 747.256935s 2024-09-13T11:52:46.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.944+0000 7fdbb565c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fdbb013d030 0x7fdbb013f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:46.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.944+0000 7fdbb565c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fdbb013d030 0x7fdbb013f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:50060/0 (socket says 172.21.15.122:50060) 2024-09-13T11:52:46.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.944+0000 7fdbb565c640 1 -- 172.21.15.122:0/2425611845 learned_addr learned my addr 172.21.15.122:0/2425611845 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-09-13T11:52:46.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.944+0000 7fdbb565c640 10 monclient(hunting): get_auth_request con 0x7fdbb013d030 auth_method 0 2024-09-13T11:52:46.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.944+0000 7fdbb565c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:46.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.944+0000 7fdbb565c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:46.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.944+0000 7fdbb565c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:46.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.944+0000 7fdbb565c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:46.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.944+0000 7fdbb565c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:46.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.944+0000 7fdbb565c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:46.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.944+0000 7fdba7fff640 1 --2- 172.21.15.122:0/2425611845 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdbb006f540 0x7fdbb013caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:46.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.944+0000 7fdba7fff640 10 monclient(hunting): get_auth_request con 0x7fdbb006f540 auth_method 0 2024-09-13T11:52:46.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.944+0000 7fdba7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:46.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.944+0000 7fdba7fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:46.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.944+0000 7fdba7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:46.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.944+0000 7fdbb565c640 10 monclient(hunting): handle_auth_done global_id 4108 payload 274 2024-09-13T11:52:46.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.944+0000 7fdbb565c640 1 -- 172.21.15.122:0/2425611845 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdbb006f540 msgr2=0x7fdbb013caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:52:46.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.944+0000 7fdbb565c640 1 --2- 172.21.15.122:0/2425611845 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdbb006f540 0x7fdbb013caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.944+0000 7fdbb565c640 1 -- 172.21.15.122:0/2425611845 >> v1:172.21.15.100:6789/0 conn(0x7fdbb006ea50 legacy=0x7fdbb006ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:52:46.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.944+0000 7fdbb565c640 10 monclient: _finish_hunting 0 2024-09-13T11:52:46.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.944+0000 7fdbb565c640 1 monclient: found mon.noname-c 2024-09-13T11:52:46.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.944+0000 7fdbb565c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:52:46.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.944+0000 7fdbb565c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.122:3300/0 2024-09-13T11:52:46.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.944+0000 7fdbb565c640 1 -- 172.21.15.122:0/2425611845 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbb00a65c0 con 0x7fdbb013d030 2024-09-13T11:52:46.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdbb565c640 1 --2- 172.21.15.122:0/2425611845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbb013d030 0x7fdbb013f420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdba8009e50 tx=0x7fdba802a550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=404dc69de31f4a95 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:52:46.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdba77fe640 1 -- 172.21.15.122:0/2425611845 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba8036070 con 0x7fdbb013d030 2024-09-13T11:52:46.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdba77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:46.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdba77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:52:46.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdba77fe640 10 monclient: dump: 2024-09-13T11:52:46.958 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-09-13T11:52:46.958 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:46.958 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.958 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.958 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:46.958 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:46.959 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:46.959 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:46.959 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:46.959 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:46.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdba77fe640 10 monclient: _finish_auth 0 2024-09-13T11:52:46.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdba77fe640 10 monclient: _check_auth_tickets 2024-09-13T11:52:46.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdba77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:52:46.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdba77fe640 1 -- 172.21.15.122:0/2425611845 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdba8031600 con 0x7fdbb013d030 2024-09-13T11:52:46.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdbb70bf640 5 monclient: authenticate success, global_id 4108 2024-09-13T11:52:46.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdbb70bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:52:46.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdba77fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:52:46.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdbb70bf640 10 monclient: get_monmap_and_config success 2024-09-13T11:52:46.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdba77fe640 1 -- 172.21.15.122:0/2425611845 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba80319d0 con 0x7fdbb013d030 2024-09-13T11:52:46.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdbb70bf640 4 set_mon_vals no callback set 2024-09-13T11:52:46.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdba77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:46.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdba77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:52:46.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdba77fe640 10 monclient: dump: 2024-09-13T11:52:46.960 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-09-13T11:52:46.960 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:46.960 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.960 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.960 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:46.960 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:46.960 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:46.960 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:46.960 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:46.960 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:46.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdbb70bf640 10 monclient: shutdown 2024-09-13T11:52:46.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdbb70bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:52:46.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdbb70bf640 1 -- 172.21.15.122:0/2425611845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbb013d030 msgr2=0x7fdbb013f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:46.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdbb70bf640 1 --2- 172.21.15.122:0/2425611845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbb013d030 0x7fdbb013f420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdba8009e50 tx=0x7fdba802a550 comp rx=0 tx=0).stop 2024-09-13T11:52:46.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdbb70bf640 1 -- 172.21.15.122:0/2425611845 shutdown_connections 2024-09-13T11:52:46.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdbb70bf640 1 --2- 172.21.15.122:0/2425611845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbb013d030 0x7fdbb013f420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdbb70bf640 1 --2- 172.21.15.122:0/2425611845 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdbb006f540 0x7fdbb013caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdbb70bf640 1 -- 172.21.15.122:0/2425611845 >> 172.21.15.122:0/2425611845 conn(0x7fdbb00a6b50 msgr2=0x7fdbb00a6f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:52:46.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdbb70bf640 1 -- 172.21.15.122:0/2425611845 shutdown_connections 2024-09-13T11:52:46.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.945+0000 7fdbb70bf640 1 -- 172.21.15.122:0/2425611845 wait complete. 2024-09-13T11:52:46.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.946+0000 7fdbb70bf640 10 monclient: build_initial_monmap 2024-09-13T11:52:46.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.946+0000 7fdbb70bf640 10 monclient: monmap: 2024-09-13T11:52:46.961 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 0 2024-09-13T11:52:46.961 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:52:46.961 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 0.000000 2024-09-13T11:52:46.962 INFO:teuthology.orchestra.run.smithi122.stderr:created 0.000000 2024-09-13T11:52:46.962 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 0 (unknown) 2024-09-13T11:52:46.962 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:46.962 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:52:46.962 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:52:46.962 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:52:46.962 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:46.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.946+0000 7fdbb70bf640 1 Processor -- start 2024-09-13T11:52:46.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.946+0000 7fdbb70bf640 1 -- start start 2024-09-13T11:52:46.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.946+0000 7fdbb70bf640 10 monclient: init 2024-09-13T11:52:46.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb70bf640 10 monclient: _reopen_session rank -1 2024-09-13T11:52:46.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb70bf640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:52:46.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb70bf640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdbb006ea50 0x7fdbb01cc620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:46.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb70bf640 10 monclient(hunting): picked mon.noname-c con 0x7fdbb006ea50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:52:46.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb70bf640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdbb006f540 0x7fdbb0135af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:46.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb70bf640 10 monclient(hunting): picked mon.noname-a con 0x7fdbb006f540 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:52:46.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb70bf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbb013d030 0x7fdbb0136030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:46.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb70bf640 10 monclient(hunting): picked mon.noname-b con 0x7fdbb013d030 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:52:46.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb70bf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:46.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb70bf640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb00a5c70 con 0x7fdbb006f540 2024-09-13T11:52:46.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb70bf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:46.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb70bf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb00a5af0 con 0x7fdbb013d030 2024-09-13T11:52:46.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb70bf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:46.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb70bf640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fdbb00a27e0 con 0x7fdbb006ea50 2024-09-13T11:52:46.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb70bf640 10 monclient(hunting): _renew_subs 2024-09-13T11:52:46.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb70bf640 10 monclient(hunting): authenticate will time out at 747.260292s 2024-09-13T11:52:46.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbb013d030 0x7fdbb0136030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:46.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbb013d030 0x7fdbb0136030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:50068/0 (socket says 172.21.15.122:50068) 2024-09-13T11:52:46.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 1 -- 172.21.15.122:0/2325380089 learned_addr learned my addr 172.21.15.122:0/2325380089 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-09-13T11:52:46.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 10 monclient(hunting): get_auth_request con 0x7fdbb013d030 auth_method 0 2024-09-13T11:52:46.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:46.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:46.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:46.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:46.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:46.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:46.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 10 monclient(hunting): handle_auth_done global_id 4111 payload 824 2024-09-13T11:52:46.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdba7fff640 1 --2- 172.21.15.122:0/2325380089 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdbb006f540 0x7fdbb0135af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:46.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb4e5b640 1 --2- 172.21.15.122:0/2325380089 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdbb006ea50 0x7fdbb01cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:46.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 1 -- 172.21.15.122:0/2325380089 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdbb006ea50 msgr2=0x7fdbb01cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:46.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 1 --2- 172.21.15.122:0/2325380089 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdbb006ea50 0x7fdbb01cc620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 1 -- 172.21.15.122:0/2325380089 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdbb006f540 msgr2=0x7fdbb0135af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:46.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 1 --2- 172.21.15.122:0/2325380089 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdbb006f540 0x7fdbb0135af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 10 monclient: _finish_hunting 0 2024-09-13T11:52:46.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 1 monclient: found mon.noname-b 2024-09-13T11:52:46.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:52:46.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.122:3300/0 2024-09-13T11:52:46.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 1 -- 172.21.15.122:0/2325380089 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbb00a8e10 con 0x7fdbb013d030 2024-09-13T11:52:46.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 1 --2- 172.21.15.122:0/2325380089 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbb013d030 0x7fdbb0136030 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdba8009e50 tx=0x7fdba8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:52:46.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 1 -- 172.21.15.122:0/2325380089 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbb013d030 msgr2=0x7fdbb0136030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-13T11:52:46.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 1 -- 172.21.15.122:0/2325380089 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbb013d030 msgr2=0x7fdbb0136030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T11:52:46.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 1 --2- 172.21.15.122:0/2325380089 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbb013d030 0x7fdbb0136030 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdba8009e50 tx=0x7fdba8004a40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:52:46.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdba57fa640 1 -- 172.21.15.122:0/2325380089 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba8036050 con 0x7fdbb013d030 2024-09-13T11:52:46.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdbb565c640 1 --2- 172.21.15.122:0/2325380089 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbb013d030 0x7fdbb0136030 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdba8009e50 tx=0x7fdba8004a40 comp rx=0 tx=0).stop 2024-09-13T11:52:46.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdba57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:46.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdba57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:52:46.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdba57fa640 10 monclient: dump: 2024-09-13T11:52:46.966 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-09-13T11:52:46.966 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:46.966 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.966 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.966 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:46.967 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:46.967 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:46.967 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:46.967 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:46.967 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:46.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdba57fa640 10 monclient: _finish_auth 0 2024-09-13T11:52:46.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdba57fa640 10 monclient: _check_auth_tickets 2024-09-13T11:52:46.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdba57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:52:46.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdba57fa640 10 monclient: ms_handle_reset current mon [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:52:46.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdba57fa640 10 monclient: _reopen_session rank -1 2024-09-13T11:52:46.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdba57fa640 1 -- 172.21.15.122:0/2325380089 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbb013d030 msgr2=0x7fdbb0136030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:52:46.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdba57fa640 1 --2- 172.21.15.122:0/2325380089 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbb013d030 0x7fdbb0136030 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdba57fa640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:52:46.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdba57fa640 1 --2- 172.21.15.122:0/2325380089 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb9c0036e0 0x7fdb9c005b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:46.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdba57fa640 10 monclient(hunting): picked mon.b con 0x7fdb9c0036e0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:52:46.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdba57fa640 1 --2- 172.21.15.122:0/2325380089 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdbb006ea50 0x7fdb9c008630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:46.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.947+0000 7fdba57fa640 10 monclient(hunting): picked mon.c con 0x7fdbb006ea50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:52:46.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba57fa640 1 --2- 172.21.15.122:0/2325380089 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdbb006f540 0x7fdb9c00b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:46.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba57fa640 10 monclient(hunting): picked mon.a con 0x7fdbb006f540 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:52:46.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba57fa640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:46.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba57fa640 1 -- 172.21.15.122:0/2325380089 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9c00bb10 con 0x7fdbb006f540 2024-09-13T11:52:46.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba57fa640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:46.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba57fa640 1 -- 172.21.15.122:0/2325380089 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9c00bc90 con 0x7fdb9c0036e0 2024-09-13T11:52:46.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba57fa640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:46.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba57fa640 1 -- 172.21.15.122:0/2325380089 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fdb9c00be60 con 0x7fdbb006ea50 2024-09-13T11:52:46.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba57fa640 10 monclient(hunting): _renew_subs 2024-09-13T11:52:46.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb4e5b640 1 --2- 172.21.15.122:0/2325380089 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb9c0036e0 0x7fdb9c005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:46.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb4e5b640 10 monclient(hunting): get_auth_request con 0x7fdb9c0036e0 auth_method 0 2024-09-13T11:52:46.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb4e5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:46.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb4e5b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:46.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb4e5b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:52:46.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb565c640 1 --2- 172.21.15.122:0/2325380089 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdbb006f540 0x7fdb9c00b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:46.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba7fff640 1 --2- 172.21.15.122:0/2325380089 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdbb006ea50 0x7fdb9c008630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:46.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba7fff640 10 monclient(hunting): get_auth_request con 0x7fdbb006ea50 auth_method 0 2024-09-13T11:52:46.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:46.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba7fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:46.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:52:46.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb4e5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:46.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb4e5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:46.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb4e5b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:52:46.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb565c640 10 monclient(hunting): get_auth_request con 0x7fdbb006f540 auth_method 0 2024-09-13T11:52:46.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb565c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:46.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb565c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:46.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb565c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:52:46.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:46.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:46.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:52:46.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb565c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:46.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb565c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:46.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb565c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:52:46.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb4e5b640 10 monclient(hunting): handle_auth_done global_id 4111 payload 293 2024-09-13T11:52:46.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb4e5b640 1 -- 172.21.15.122:0/2325380089 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdbb006ea50 msgr2=0x7fdb9c008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:46.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb4e5b640 1 --2- 172.21.15.122:0/2325380089 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdbb006ea50 0x7fdb9c008630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb4e5b640 1 -- 172.21.15.122:0/2325380089 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdbb006f540 msgr2=0x7fdb9c00b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:46.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb4e5b640 1 --2- 172.21.15.122:0/2325380089 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdbb006f540 0x7fdb9c00b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:46.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb4e5b640 10 monclient: _finish_hunting 0 2024-09-13T11:52:46.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb4e5b640 1 monclient: found mon.b 2024-09-13T11:52:46.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb4e5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:52:46.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb4e5b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:52:46.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb4e5b640 1 -- 172.21.15.122:0/2325380089 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=2+}) -- 0x7fdb9c00c060 con 0x7fdb9c0036e0 2024-09-13T11:52:46.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb4e5b640 10 monclient: _finish_auth 0 2024-09-13T11:52:46.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb4e5b640 10 monclient: _check_auth_tickets 2024-09-13T11:52:46.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb4e5b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:52:46.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb70bf640 5 monclient: authenticate success, global_id 4111 2024-09-13T11:52:46.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb70bf640 10 monclient: _renew_subs 2024-09-13T11:52:46.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb70bf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:52:46.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb70bf640 1 -- 172.21.15.122:0/2325380089 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbb00a10d0 con 0x7fdb9c0036e0 2024-09-13T11:52:46.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb70bf640 10 monclient: _renew_subs 2024-09-13T11:52:46.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb70bf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:52:46.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb70bf640 1 -- 172.21.15.122:0/2325380089 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbb01224a0 con 0x7fdb9c0036e0 2024-09-13T11:52:46.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdbb4e5b640 1 --2- 172.21.15.122:0/2325380089 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb9c0036e0 0x7fdb9c005b10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00a7e0 tx=0x7fdba8009000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:52:46.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba57fa640 1 -- 172.21.15.122:0/2325380089 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbac015070 con 0x7fdb9c0036e0 2024-09-13T11:52:46.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:46.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:52:46.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba57fa640 10 monclient: dump: 2024-09-13T11:52:46.973 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-09-13T11:52:46.973 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:46.973 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.973 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:46.973 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:46.973 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:46.973 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:46.973 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:46.973 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:46.973 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:46.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba57fa640 1 -- 172.21.15.122:0/2325380089 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbac00ae20 con 0x7fdb9c0036e0 2024-09-13T11:52:46.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba57fa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:52:46.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba6ffd640 4 set_mon_vals no callback set 2024-09-13T11:52:46.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba57fa640 1 -- 172.21.15.122:0/2325380089 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdbac004020 con 0x7fdb9c0036e0 2024-09-13T11:52:46.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.948+0000 7fdba57fa640 1 -- 172.21.15.122:0/2325380089 <== mon.1 v2:172.21.15.122:3300/0 4 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fdbac003d60 con 0x7fdb9c0036e0 2024-09-13T11:52:46.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.949+0000 7fdbb70bf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:46.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.949+0000 7fdbb70bf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:46.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.949+0000 7fdbb70bf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:52:46.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.949+0000 7fdbb70bf640 1 -- 172.21.15.122:0/2325380089 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb90005080 con 0x7fdb9c0036e0 2024-09-13T11:52:46.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.951+0000 7fdba57fa640 1 -- 172.21.15.122:0/2325380089 <== mon.1 v2:172.21.15.122:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fdbac01e030 con 0x7fdb9c0036e0 2024-09-13T11:52:46.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.951+0000 7fdba57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:46.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:46.951+0000 7fdba57fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:52:47.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.020+0000 7fdbb70bf640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8d8d39e7-e8f1-499a-a01b-e7ba49844eae", "id": 1}] 2024-09-13T11:52:47.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.020+0000 7fdbb70bf640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8d8d39e7-e8f1-499a-a01b-e7ba49844eae", "id": 1}] 2024-09-13T11:52:47.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.020+0000 7fdbb70bf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:52:47.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.020+0000 7fdbb70bf640 1 -- 172.21.15.122:0/2325380089 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8d8d39e7-e8f1-499a-a01b-e7ba49844eae", "id": 1} v 0) -- 0x7fdb90005670 con 0x7fdb9c0036e0 2024-09-13T11:52:47.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.028+0000 7fdba57fa640 1 -- 172.21.15.122:0/2325380089 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8d8d39e7-e8f1-499a-a01b-e7ba49844eae", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fdbac00c6d0 con 0x7fdb9c0036e0 2024-09-13T11:52:47.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.028+0000 7fdba57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8d8d39e7-e8f1-499a-a01b-e7ba49844eae", "id": 1}] 2024-09-13T11:52:47.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.028+0000 7fdba57fa640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:52:47.030 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-09-13T11:52:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.029+0000 7fdbb70bf640 10 monclient: shutdown 2024-09-13T11:52:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.029+0000 7fdbb70bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:52:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.029+0000 7fdbb70bf640 1 -- 172.21.15.122:0/2325380089 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb9c0036e0 msgr2=0x7fdb9c005b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.029+0000 7fdbb70bf640 1 --2- 172.21.15.122:0/2325380089 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb9c0036e0 0x7fdb9c005b10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00a7e0 tx=0x7fdba8009000 comp rx=0 tx=0).stop 2024-09-13T11:52:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.029+0000 7fdbb70bf640 1 -- 172.21.15.122:0/2325380089 shutdown_connections 2024-09-13T11:52:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.029+0000 7fdbb70bf640 1 --2- 172.21.15.122:0/2325380089 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdbb006f540 0x7fdb9c00b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.029+0000 7fdbb70bf640 1 --2- 172.21.15.122:0/2325380089 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdbb006ea50 0x7fdb9c008630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.029+0000 7fdbb70bf640 1 --2- 172.21.15.122:0/2325380089 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb9c0036e0 0x7fdb9c005b10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.029+0000 7fdbb70bf640 1 -- 172.21.15.122:0/2325380089 >> 172.21.15.122:0/2325380089 conn(0x7fdbb00a6b50 msgr2=0x7fdbb0136740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:52:47.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.029+0000 7fdbb70bf640 1 -- 172.21.15.122:0/2325380089 shutdown_connections 2024-09-13T11:52:47.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.029+0000 7fdbb70bf640 1 -- 172.21.15.122:0/2325380089 wait complete. 2024-09-13T11:52:47.042 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new e0bd3e30-0ff3-4067-83d3-ef8d84b29097 2 2024-09-13T11:52:47.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.137+0000 7f3a34f24640 10 monclient: get_monmap_and_config 2024-09-13T11:52:47.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.137+0000 7f3a34f24640 10 monclient: build_initial_monmap 2024-09-13T11:52:47.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.137+0000 7f3a34f24640 10 monclient: monmap: 2024-09-13T11:52:47.139 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 0 2024-09-13T11:52:47.139 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.139 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:47.138822+0000 2024-09-13T11:52:47.139 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:47.138822+0000 2024-09-13T11:52:47.140 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 0 (unknown) 2024-09-13T11:52:47.140 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:47.140 INFO:teuthology.orchestra.run.smithi122.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:52:47.140 INFO:teuthology.orchestra.run.smithi122.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:52:47.140 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:52:47.140 INFO:teuthology.orchestra.run.smithi122.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:52:47.140 INFO:teuthology.orchestra.run.smithi122.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:52:47.140 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:47.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.137+0000 7f3a34f24640 1 Processor -- start 2024-09-13T11:52:47.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.138+0000 7f3a34f24640 1 -- start start 2024-09-13T11:52:47.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.138+0000 7f3a34f24640 10 monclient: init 2024-09-13T11:52:47.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.138+0000 7f3a34f24640 10 monclient: _reopen_session rank -1 2024-09-13T11:52:47.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.138+0000 7f3a34f24640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-13T11:52:47.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.138+0000 7f3a34f24640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3a3006ea50 addr v1:172.21.15.122:6789/0 2024-09-13T11:52:47.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.138+0000 7f3a34f24640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a3006f540 addr v1:172.21.15.100:6789/0 2024-09-13T11:52:47.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.138+0000 7f3a34f24640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3a300a2830 0x7f3a30140220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.138+0000 7f3a34f24640 10 monclient(hunting): picked mon.noname-c con 0x7f3a300a2830 addr v2:172.21.15.122:3300/0 2024-09-13T11:52:47.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.138+0000 7f3a34f24640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a300a6440 con 0x7f3a3006f540 2024-09-13T11:52:47.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.138+0000 7f3a34f24640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a300a6680 con 0x7f3a3006ea50 2024-09-13T11:52:47.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.138+0000 7f3a34f24640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.138+0000 7f3a34f24640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f3a300a2ca0 con 0x7f3a300a2830 2024-09-13T11:52:47.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.138+0000 7f3a34f24640 10 monclient(hunting): _renew_subs 2024-09-13T11:52:47.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.138+0000 7f3a34f24640 10 monclient(hunting): authenticate will time out at 747.451738s 2024-09-13T11:52:47.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.138+0000 7f3a2e59c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f3a3006ea50 0x7f3a3006ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:56266/0 (socket says 172.21.15.122:56266) 2024-09-13T11:52:47.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.138+0000 7f3a2ed9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3a300a2830 0x7f3a30140220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.138+0000 7f3a2e59c640 1 -- 172.21.15.122:0/2836904203 learned_addr learned my addr 172.21.15.122:0/2836904203 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-09-13T11:52:47.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.138+0000 7f3a2ed9d640 10 monclient(hunting): get_auth_request con 0x7f3a300a2830 auth_method 0 2024-09-13T11:52:47.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.138+0000 7f3a2ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:47.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.138+0000 7f3a2ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:47.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.138+0000 7f3a2ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:47.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.138+0000 7f3a2ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:47.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.138+0000 7f3a2ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:47.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.138+0000 7f3a2ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:47.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.139+0000 7f3a2ed9d640 10 monclient(hunting): handle_auth_done global_id 4117 payload 274 2024-09-13T11:52:47.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.139+0000 7f3a2ed9d640 1 -- 172.21.15.122:0/2836904203 >> v1:172.21.15.122:6789/0 conn(0x7f3a3006ea50 legacy=0x7f3a3006ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:47.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.139+0000 7f3a2ed9d640 1 -- 172.21.15.122:0/2836904203 >> v1:172.21.15.100:6789/0 conn(0x7f3a3006f540 legacy=0x7f3a3013caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:47.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.139+0000 7f3a2ed9d640 10 monclient: _finish_hunting 0 2024-09-13T11:52:47.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.139+0000 7f3a2ed9d640 1 monclient: found mon.noname-c 2024-09-13T11:52:47.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.139+0000 7f3a2ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:52:47.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.139+0000 7f3a2ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.122:3300/0 2024-09-13T11:52:47.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.139+0000 7f3a2ed9d640 1 -- 172.21.15.122:0/2836904203 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a3006fc60 con 0x7f3a300a2830 2024-09-13T11:52:47.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.139+0000 7f3a2ed9d640 1 --2- 172.21.15.122:0/2836904203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a300a2830 0x7f3a30140220 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009d90 tx=0x7f3a2802a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=824da2dce4ca3a70 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:52:47.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.139+0000 7f3a2d59a640 1 -- 172.21.15.122:0/2836904203 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a28031600 con 0x7f3a300a2830 2024-09-13T11:52:47.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.139+0000 7f3a2d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:47.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.139+0000 7f3a2d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:52:47.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.139+0000 7f3a2d59a640 10 monclient: dump: 2024-09-13T11:52:47.149 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-09-13T11:52:47.149 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.150 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.150 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.150 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:47.150 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:47.150 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:47.150 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:47.150 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:47.150 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:47.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.139+0000 7f3a2d59a640 10 monclient: _finish_auth 0 2024-09-13T11:52:47.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.139+0000 7f3a2d59a640 10 monclient: _check_auth_tickets 2024-09-13T11:52:47.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.139+0000 7f3a2d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:52:47.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.139+0000 7f3a2d59a640 1 -- 172.21.15.122:0/2836904203 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a280317a0 con 0x7f3a300a2830 2024-09-13T11:52:47.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.139+0000 7f3a34f24640 5 monclient: authenticate success, global_id 4117 2024-09-13T11:52:47.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.139+0000 7f3a34f24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:52:47.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.139+0000 7f3a2d59a640 10 monclient: handle_config config(0 keys) 2024-09-13T11:52:47.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.140+0000 7f3a34f24640 10 monclient: get_monmap_and_config success 2024-09-13T11:52:47.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.140+0000 7f3a2d59a640 1 -- 172.21.15.122:0/2836904203 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a28031b70 con 0x7f3a300a2830 2024-09-13T11:52:47.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.140+0000 7f3a2d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:47.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.140+0000 7f3a34f24640 4 set_mon_vals no callback set 2024-09-13T11:52:47.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.140+0000 7f3a2d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:52:47.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.140+0000 7f3a2d59a640 10 monclient: dump: 2024-09-13T11:52:47.151 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-09-13T11:52:47.151 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.151 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.151 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.151 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:47.152 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:47.152 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:47.152 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:47.152 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:47.152 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:47.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.140+0000 7f3a34f24640 10 monclient: shutdown 2024-09-13T11:52:47.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.140+0000 7f3a34f24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:52:47.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.140+0000 7f3a34f24640 1 -- 172.21.15.122:0/2836904203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a300a2830 msgr2=0x7f3a30140220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:47.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.140+0000 7f3a34f24640 1 --2- 172.21.15.122:0/2836904203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a300a2830 0x7f3a30140220 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009d90 tx=0x7f3a2802a5d0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.140+0000 7f3a34f24640 1 -- 172.21.15.122:0/2836904203 shutdown_connections 2024-09-13T11:52:47.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.140+0000 7f3a34f24640 1 --2- 172.21.15.122:0/2836904203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a300a2830 0x7f3a30140220 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.140+0000 7f3a34f24640 1 -- 172.21.15.122:0/2836904203 >> 172.21.15.122:0/2836904203 conn(0x7f3a300a6970 msgr2=0x7f3a300a6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:52:47.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.140+0000 7f3a34f24640 1 -- 172.21.15.122:0/2836904203 shutdown_connections 2024-09-13T11:52:47.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.140+0000 7f3a34f24640 1 -- 172.21.15.122:0/2836904203 wait complete. 2024-09-13T11:52:47.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.141+0000 7f3a34f24640 10 monclient: build_initial_monmap 2024-09-13T11:52:47.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.141+0000 7f3a34f24640 10 monclient: monmap: 2024-09-13T11:52:47.153 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 0 2024-09-13T11:52:47.153 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:52:47.153 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 0.000000 2024-09-13T11:52:47.153 INFO:teuthology.orchestra.run.smithi122.stderr:created 0.000000 2024-09-13T11:52:47.153 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 0 (unknown) 2024-09-13T11:52:47.153 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:47.153 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:52:47.153 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:52:47.153 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:52:47.153 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:47.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.141+0000 7f3a34f24640 1 Processor -- start 2024-09-13T11:52:47.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.141+0000 7f3a34f24640 1 -- start start 2024-09-13T11:52:47.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.141+0000 7f3a34f24640 10 monclient: init 2024-09-13T11:52:47.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a34f24640 10 monclient: _reopen_session rank -1 2024-09-13T11:52:47.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a34f24640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:52:47.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a34f24640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a3006ea50 0x7f3a30137230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a34f24640 10 monclient(hunting): picked mon.noname-a con 0x7f3a3006ea50 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:52:47.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a34f24640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a3006f540 0x7f3a30137770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a34f24640 10 monclient(hunting): picked mon.noname-c con 0x7f3a3006f540 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:52:47.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a34f24640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a300a2830 0x7f3a3013ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a34f24640 10 monclient(hunting): picked mon.noname-b con 0x7f3a300a2830 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:52:47.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a34f24640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a34f24640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f3a30140760 con 0x7f3a3006ea50 2024-09-13T11:52:47.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a34f24640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a34f24640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f3a300a1e50 con 0x7f3a300a2830 2024-09-13T11:52:47.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a34f24640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a34f24640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f3a301408e0 con 0x7f3a3006f540 2024-09-13T11:52:47.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a34f24640 10 monclient(hunting): _renew_subs 2024-09-13T11:52:47.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a34f24640 10 monclient(hunting): authenticate will time out at 747.455325s 2024-09-13T11:52:47.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a300a2830 0x7f3a3013ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a300a2830 0x7f3a3013ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:50110/0 (socket says 172.21.15.122:50110) 2024-09-13T11:52:47.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2ed9d640 1 -- 172.21.15.122:0/2694176711 learned_addr learned my addr 172.21.15.122:0/2694176711 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-09-13T11:52:47.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2ed9d640 10 monclient(hunting): get_auth_request con 0x7f3a300a2830 auth_method 0 2024-09-13T11:52:47.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:47.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:47.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:47.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2dd9b640 1 --2- 172.21.15.122:0/2694176711 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a3006f540 0x7f3a30137770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2dd9b640 10 monclient(hunting): get_auth_request con 0x7f3a3006f540 auth_method 0 2024-09-13T11:52:47.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:47.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:47.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:47.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2e59c640 1 --2- 172.21.15.122:0/2694176711 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a3006ea50 0x7f3a30137230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2e59c640 10 monclient(hunting): get_auth_request con 0x7f3a3006ea50 auth_method 0 2024-09-13T11:52:47.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:47.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2e59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:47.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:47.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:47.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:47.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:47.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:47.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:47.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:47.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:47.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:47.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:47.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2dd9b640 1 -- 172.21.15.122:0/2694176711 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a3006f540 msgr2=0x7f3a30137770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-13T11:52:47.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2dd9b640 1 -- 172.21.15.122:0/2694176711 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a3006f540 msgr2=0x7f3a30137770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T11:52:47.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2dd9b640 1 --2- 172.21.15.122:0/2694176711 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a3006f540 0x7f3a30137770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:52:47.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2dd9b640 1 --2- 172.21.15.122:0/2694176711 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a3006f540 0x7f3a30137770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:52:47.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2e59c640 10 monclient(hunting): handle_auth_done global_id 4116 payload 824 2024-09-13T11:52:47.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2e59c640 1 -- 172.21.15.122:0/2694176711 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a3006f540 msgr2=0x7f3a30137770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:52:47.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2e59c640 1 --2- 172.21.15.122:0/2694176711 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a3006f540 0x7f3a30137770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2e59c640 1 -- 172.21.15.122:0/2694176711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a300a2830 msgr2=0x7f3a3013ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:47.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2e59c640 1 --2- 172.21.15.122:0/2694176711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a300a2830 0x7f3a3013ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2e59c640 10 monclient: _finish_hunting 0 2024-09-13T11:52:47.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2e59c640 1 monclient: found mon.noname-a 2024-09-13T11:52:47.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:52:47.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:52:47.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2e59c640 1 -- 172.21.15.122:0/2694176711 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a3013fc40 con 0x7f3a3006ea50 2024-09-13T11:52:47.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.142+0000 7f3a2ed9d640 1 --2- 172.21.15.122:0/2694176711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a300a2830 0x7f3a3013ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:52:47.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.143+0000 7f3a2e59c640 1 --2- 172.21.15.122:0/2694176711 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a3006ea50 0x7f3a30137230 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3a20005880 tx=0x7f3a2000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:52:47.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.143+0000 7f3a1effd640 1 -- 172.21.15.122:0/2694176711 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a20017070 con 0x7f3a3006ea50 2024-09-13T11:52:47.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.143+0000 7f3a1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:47.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.143+0000 7f3a1effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:52:47.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.143+0000 7f3a1effd640 10 monclient: dump: 2024-09-13T11:52:47.158 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-09-13T11:52:47.158 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.158 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.159 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.159 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:47.159 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:47.159 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:47.159 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:47.159 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:47.159 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:47.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.143+0000 7f3a1effd640 10 monclient: _finish_auth 0 2024-09-13T11:52:47.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.143+0000 7f3a1effd640 10 monclient: _check_auth_tickets 2024-09-13T11:52:47.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.143+0000 7f3a1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:52:47.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.143+0000 7f3a1effd640 1 -- 172.21.15.122:0/2694176711 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a20005ce0 con 0x7f3a3006ea50 2024-09-13T11:52:47.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.143+0000 7f3a34f24640 5 monclient: authenticate success, global_id 4116 2024-09-13T11:52:47.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.143+0000 7f3a1effd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:52:47.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.143+0000 7f3a1effd640 1 -- 172.21.15.122:0/2694176711 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a200072f0 con 0x7f3a3006ea50 2024-09-13T11:52:47.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.143+0000 7f3a34f24640 10 monclient: _renew_subs 2024-09-13T11:52:47.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.143+0000 7f3a34f24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:52:47.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.143+0000 7f3a34f24640 1 -- 172.21.15.122:0/2694176711 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a300a9610 con 0x7f3a3006ea50 2024-09-13T11:52:47.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.143+0000 7f3a1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:47.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.143+0000 7f3a1effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:52:47.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.143+0000 7f3a1effd640 10 monclient: dump: 2024-09-13T11:52:47.160 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-09-13T11:52:47.160 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.160 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.160 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.160 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:47.161 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:47.161 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:47.161 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:47.161 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:47.161 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:47.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.143+0000 7f3a2cd99640 4 set_mon_vals no callback set 2024-09-13T11:52:47.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.143+0000 7f3a34f24640 10 monclient: _renew_subs 2024-09-13T11:52:47.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.143+0000 7f3a34f24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:52:47.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.143+0000 7f3a34f24640 1 -- 172.21.15.122:0/2694176711 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a30120df0 con 0x7f3a3006ea50 2024-09-13T11:52:47.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.143+0000 7f3a1effd640 1 -- 172.21.15.122:0/2694176711 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3a2001a070 con 0x7f3a3006ea50 2024-09-13T11:52:47.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.143+0000 7f3a1effd640 1 -- 172.21.15.122:0/2694176711 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f3a20012820 con 0x7f3a3006ea50 2024-09-13T11:52:47.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.144+0000 7f3a34f24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:47.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.144+0000 7f3a34f24640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:47.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.144+0000 7f3a34f24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:52:47.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.144+0000 7f3a34f24640 1 -- 172.21.15.122:0/2694176711 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a3006ee30 con 0x7f3a3006ea50 2024-09-13T11:52:47.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.146+0000 7f3a1effd640 1 -- 172.21.15.122:0/2694176711 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f3a20021070 con 0x7f3a3006ea50 2024-09-13T11:52:47.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.146+0000 7f3a1effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:47.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.146+0000 7f3a1effd640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:52:47.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.217+0000 7f3a34f24640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e0bd3e30-0ff3-4067-83d3-ef8d84b29097", "id": 2}] 2024-09-13T11:52:47.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.217+0000 7f3a34f24640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e0bd3e30-0ff3-4067-83d3-ef8d84b29097", "id": 2}] 2024-09-13T11:52:47.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.217+0000 7f3a34f24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:52:47.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.217+0000 7f3a34f24640 1 -- 172.21.15.122:0/2694176711 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e0bd3e30-0ff3-4067-83d3-ef8d84b29097", "id": 2} v 0) -- 0x7f3a301208f0 con 0x7f3a3006ea50 2024-09-13T11:52:47.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.219+0000 7f3a1effd640 1 -- 172.21.15.122:0/2694176711 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e0bd3e30-0ff3-4067-83d3-ef8d84b29097", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f3a300a9610 con 0x7f3a3006ea50 2024-09-13T11:52:47.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.219+0000 7f3a1effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e0bd3e30-0ff3-4067-83d3-ef8d84b29097", "id": 2}] 2024-09-13T11:52:47.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.219+0000 7f3a1effd640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:52:47.221 INFO:teuthology.orchestra.run.smithi122.stdout:2 2024-09-13T11:52:47.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.220+0000 7f3a34f24640 10 monclient: shutdown 2024-09-13T11:52:47.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.220+0000 7f3a34f24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:52:47.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.220+0000 7f3a34f24640 1 -- 172.21.15.122:0/2694176711 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a3006ea50 msgr2=0x7f3a30137230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:47.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.220+0000 7f3a34f24640 1 --2- 172.21.15.122:0/2694176711 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a3006ea50 0x7f3a30137230 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3a20005880 tx=0x7f3a2000c960 comp rx=0 tx=0).stop 2024-09-13T11:52:47.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.220+0000 7f3a34f24640 1 -- 172.21.15.122:0/2694176711 shutdown_connections 2024-09-13T11:52:47.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.220+0000 7f3a34f24640 1 --2- 172.21.15.122:0/2694176711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a300a2830 0x7f3a3013ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.220+0000 7f3a34f24640 1 --2- 172.21.15.122:0/2694176711 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a3006f540 0x7f3a30137770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.220+0000 7f3a34f24640 1 --2- 172.21.15.122:0/2694176711 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a3006ea50 0x7f3a30137230 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.220+0000 7f3a34f24640 1 -- 172.21.15.122:0/2694176711 >> 172.21.15.122:0/2694176711 conn(0x7f3a300a6970 msgr2=0x7f3a300a7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:52:47.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.220+0000 7f3a34f24640 1 -- 172.21.15.122:0/2694176711 shutdown_connections 2024-09-13T11:52:47.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.220+0000 7f3a34f24640 1 -- 172.21.15.122:0/2694176711 wait complete. 2024-09-13T11:52:47.232 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new eca4429b-38cd-488a-b31d-c5856d701bde 3 2024-09-13T11:52:47.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.331+0000 7f94c9da3640 10 monclient: get_monmap_and_config 2024-09-13T11:52:47.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.332+0000 7f94c9da3640 10 monclient: build_initial_monmap 2024-09-13T11:52:47.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.332+0000 7f94c9da3640 10 monclient: monmap: 2024-09-13T11:52:47.334 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 0 2024-09-13T11:52:47.334 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.334 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:47.333393+0000 2024-09-13T11:52:47.334 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:47.333393+0000 2024-09-13T11:52:47.334 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 0 (unknown) 2024-09-13T11:52:47.334 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:47.334 INFO:teuthology.orchestra.run.smithi122.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:52:47.334 INFO:teuthology.orchestra.run.smithi122.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:52:47.334 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:52:47.334 INFO:teuthology.orchestra.run.smithi122.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:52:47.335 INFO:teuthology.orchestra.run.smithi122.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:52:47.335 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:47.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.332+0000 7f94c9da3640 1 Processor -- start 2024-09-13T11:52:47.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.332+0000 7f94c9da3640 1 -- start start 2024-09-13T11:52:47.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.332+0000 7f94c9da3640 10 monclient: init 2024-09-13T11:52:47.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c9da3640 10 monclient: _reopen_session rank -1 2024-09-13T11:52:47.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c9da3640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-13T11:52:47.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c9da3640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f94c406ea50 0x7f94c40a2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c9da3640 10 monclient(hunting): picked mon.noname-a con 0x7f94c406ea50 addr v2:172.21.15.100:3300/0 2024-09-13T11:52:47.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c9da3640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94c406ee30 0x7f94c406f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c9da3640 10 monclient(hunting): picked mon.noname-b con 0x7f94c406ee30 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:52:47.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c9da3640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f94c4137630 0x7f94c4137aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c9da3640 10 monclient(hunting): picked mon.noname-c con 0x7f94c4137630 addr v2:172.21.15.122:3300/0 2024-09-13T11:52:47.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c9da3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c9da3640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f94c40a1e50 con 0x7f94c406ea50 2024-09-13T11:52:47.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c9da3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c9da3640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f94c40a6440 con 0x7f94c4137630 2024-09-13T11:52:47.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c9da3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c9da3640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f94c40a6680 con 0x7f94c406ee30 2024-09-13T11:52:47.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c9da3640 10 monclient(hunting): _renew_subs 2024-09-13T11:52:47.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c9da3640 10 monclient(hunting): authenticate will time out at 747.646327s 2024-09-13T11:52:47.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c3fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f94c4137630 0x7f94c4137aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c3fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f94c4137630 0x7f94c4137aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:50122/0 (socket says 172.21.15.122:50122) 2024-09-13T11:52:47.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c3fff640 1 -- 172.21.15.122:0/2995283521 learned_addr learned my addr 172.21.15.122:0/2995283521 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-09-13T11:52:47.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c3fff640 10 monclient(hunting): get_auth_request con 0x7f94c4137630 auth_method 0 2024-09-13T11:52:47.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:47.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:47.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:47.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:47.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:47.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:47.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c2ffd640 1 --2- 172.21.15.122:0/2995283521 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94c406ee30 0x7f94c406f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c37fe640 1 --2- 172.21.15.122:0/2995283521 >> v2:172.21.15.100:3300/0 conn(0x7f94c406ea50 0x7f94c40a2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c2ffd640 10 monclient(hunting): get_auth_request con 0x7f94c406ee30 auth_method 0 2024-09-13T11:52:47.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:47.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:47.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:47.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c37fe640 10 monclient(hunting): get_auth_request con 0x7f94c406ea50 auth_method 0 2024-09-13T11:52:47.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:47.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:47.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:47.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:47.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:47.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:47.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:47.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:47.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:47.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c3fff640 10 monclient(hunting): handle_auth_done global_id 4126 payload 274 2024-09-13T11:52:47.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c3fff640 1 -- 172.21.15.122:0/2995283521 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94c406ee30 msgr2=0x7f94c406f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:52:47.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c3fff640 1 --2- 172.21.15.122:0/2995283521 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94c406ee30 0x7f94c406f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c3fff640 1 -- 172.21.15.122:0/2995283521 >> v2:172.21.15.100:3300/0 conn(0x7f94c406ea50 msgr2=0x7f94c40a2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:52:47.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c3fff640 1 --2- 172.21.15.122:0/2995283521 >> v2:172.21.15.100:3300/0 conn(0x7f94c406ea50 0x7f94c40a2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c3fff640 10 monclient: _finish_hunting 0 2024-09-13T11:52:47.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c3fff640 1 monclient: found mon.noname-c 2024-09-13T11:52:47.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:52:47.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.122:3300/0 2024-09-13T11:52:47.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.333+0000 7f94c3fff640 1 -- 172.21.15.122:0/2995283521 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94c406fc60 con 0x7f94c4137630 2024-09-13T11:52:47.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.334+0000 7f94c3fff640 1 --2- 172.21.15.122:0/2995283521 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94c4137630 0x7f94c4137aa0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f94c406fba0 tx=0x7f94bc02a570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b5c542ec7c4d0b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:52:47.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.334+0000 7f94c27fc640 1 -- 172.21.15.122:0/2995283521 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc036070 con 0x7f94c4137630 2024-09-13T11:52:47.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.334+0000 7f94c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:47.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.334+0000 7f94c27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:52:47.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.334+0000 7f94c27fc640 10 monclient: dump: 2024-09-13T11:52:47.347 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-09-13T11:52:47.347 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.347 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.347 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.347 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:47.347 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:47.347 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:47.347 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:47.347 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:47.347 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:47.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.334+0000 7f94c27fc640 10 monclient: _finish_auth 0 2024-09-13T11:52:47.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.334+0000 7f94c27fc640 10 monclient: _check_auth_tickets 2024-09-13T11:52:47.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.334+0000 7f94c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:52:47.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.334+0000 7f94c9da3640 5 monclient: authenticate success, global_id 4126 2024-09-13T11:52:47.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.334+0000 7f94c9da3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:52:47.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.334+0000 7f94c27fc640 1 -- 172.21.15.122:0/2995283521 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94bc031600 con 0x7f94c4137630 2024-09-13T11:52:47.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.334+0000 7f94c27fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:52:47.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.334+0000 7f94c9da3640 10 monclient: get_monmap_and_config success 2024-09-13T11:52:47.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.334+0000 7f94c27fc640 1 -- 172.21.15.122:0/2995283521 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc0319d0 con 0x7f94c4137630 2024-09-13T11:52:47.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.334+0000 7f94c9da3640 4 set_mon_vals no callback set 2024-09-13T11:52:47.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.334+0000 7f94c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:47.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.334+0000 7f94c27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:52:47.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.334+0000 7f94c27fc640 10 monclient: dump: 2024-09-13T11:52:47.348 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-09-13T11:52:47.348 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.349 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.349 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.349 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:47.349 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:47.349 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:47.349 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:47.349 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:47.349 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:47.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.335+0000 7f94c9da3640 10 monclient: shutdown 2024-09-13T11:52:47.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.335+0000 7f94c9da3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:52:47.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.335+0000 7f94c9da3640 1 -- 172.21.15.122:0/2995283521 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94c4137630 msgr2=0x7f94c4137aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:47.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.335+0000 7f94c9da3640 1 --2- 172.21.15.122:0/2995283521 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94c4137630 0x7f94c4137aa0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f94c406fba0 tx=0x7f94bc02a570 comp rx=0 tx=0).stop 2024-09-13T11:52:47.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.335+0000 7f94c9da3640 1 -- 172.21.15.122:0/2995283521 shutdown_connections 2024-09-13T11:52:47.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.335+0000 7f94c9da3640 1 --2- 172.21.15.122:0/2995283521 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94c406ee30 0x7f94c406f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.335+0000 7f94c9da3640 1 --2- 172.21.15.122:0/2995283521 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94c4137630 0x7f94c4137aa0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.335+0000 7f94c9da3640 1 --2- 172.21.15.122:0/2995283521 >> v2:172.21.15.100:3300/0 conn(0x7f94c406ea50 0x7f94c40a2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.335+0000 7f94c9da3640 1 -- 172.21.15.122:0/2995283521 >> 172.21.15.122:0/2995283521 conn(0x7f94c40a6970 msgr2=0x7f94c40a6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:52:47.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.335+0000 7f94c9da3640 1 -- 172.21.15.122:0/2995283521 shutdown_connections 2024-09-13T11:52:47.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.335+0000 7f94c9da3640 1 -- 172.21.15.122:0/2995283521 wait complete. 2024-09-13T11:52:47.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.335+0000 7f94c9da3640 10 monclient: build_initial_monmap 2024-09-13T11:52:47.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.335+0000 7f94c9da3640 10 monclient: monmap: 2024-09-13T11:52:47.350 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 0 2024-09-13T11:52:47.350 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:52:47.350 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 0.000000 2024-09-13T11:52:47.350 INFO:teuthology.orchestra.run.smithi122.stderr:created 0.000000 2024-09-13T11:52:47.350 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 0 (unknown) 2024-09-13T11:52:47.350 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:47.351 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:52:47.351 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:52:47.351 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:52:47.351 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:47.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c9da3640 1 Processor -- start 2024-09-13T11:52:47.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c9da3640 1 -- start start 2024-09-13T11:52:47.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c9da3640 10 monclient: init 2024-09-13T11:52:47.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c9da3640 10 monclient: _reopen_session rank -1 2024-09-13T11:52:47.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c9da3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:52:47.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c9da3640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94c406ea50 0x7f94c41ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c9da3640 10 monclient(hunting): picked mon.noname-c con 0x7f94c406ea50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:52:47.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c9da3640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94c406ee30 0x7f94c41cd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c9da3640 10 monclient(hunting): picked mon.noname-a con 0x7f94c406ee30 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:52:47.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c9da3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94c4137630 0x7f94c4123f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c9da3640 10 monclient(hunting): picked mon.noname-b con 0x7f94c4137630 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:52:47.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c9da3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c9da3640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f94c40a5c70 con 0x7f94c406ee30 2024-09-13T11:52:47.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c9da3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c9da3640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f94c40a5af0 con 0x7f94c4137630 2024-09-13T11:52:47.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c9da3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c9da3640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f94c4137fe0 con 0x7f94c406ea50 2024-09-13T11:52:47.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c9da3640 10 monclient(hunting): _renew_subs 2024-09-13T11:52:47.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c9da3640 10 monclient(hunting): authenticate will time out at 747.649858s 2024-09-13T11:52:47.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94c4137630 0x7f94c4123f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94c4137630 0x7f94c4123f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:50124/0 (socket says 172.21.15.122:50124) 2024-09-13T11:52:47.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c3fff640 1 -- 172.21.15.122:0/1372285069 learned_addr learned my addr 172.21.15.122:0/1372285069 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-09-13T11:52:47.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c3fff640 10 monclient(hunting): get_auth_request con 0x7f94c4137630 auth_method 0 2024-09-13T11:52:47.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:47.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c37fe640 1 --2- 172.21.15.122:0/1372285069 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94c406ea50 0x7f94c41ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:47.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.336+0000 7f94c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:47.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c37fe640 10 monclient(hunting): get_auth_request con 0x7f94c406ea50 auth_method 0 2024-09-13T11:52:47.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:47.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:47.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:47.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c2ffd640 1 --2- 172.21.15.122:0/1372285069 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94c406ee30 0x7f94c41cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:47.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:47.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:47.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c2ffd640 10 monclient(hunting): get_auth_request con 0x7f94c406ee30 auth_method 0 2024-09-13T11:52:47.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:47.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:47.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:47.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:47.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:47.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:47.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:47.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:47.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:47.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c3fff640 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2024-09-13T11:52:47.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c3fff640 1 -- 172.21.15.122:0/1372285069 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94c406ea50 msgr2=0x7f94c41ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:47.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c3fff640 1 --2- 172.21.15.122:0/1372285069 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94c406ea50 0x7f94c41ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c3fff640 1 -- 172.21.15.122:0/1372285069 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94c406ee30 msgr2=0x7f94c41cd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:47.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c3fff640 1 --2- 172.21.15.122:0/1372285069 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94c406ee30 0x7f94c41cd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c3fff640 10 monclient: _finish_hunting 0 2024-09-13T11:52:47.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c3fff640 1 monclient: found mon.noname-b 2024-09-13T11:52:47.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:52:47.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.122:3300/0 2024-09-13T11:52:47.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c3fff640 1 -- 172.21.15.122:0/1372285069 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94c4125a10 con 0x7f94c4137630 2024-09-13T11:52:47.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c3fff640 1 --2- 172.21.15.122:0/1372285069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94c4137630 0x7f94c4123f50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009f40 tx=0x7f94bc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:52:47.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94bbfff640 1 -- 172.21.15.122:0/1372285069 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc036050 con 0x7f94c4137630 2024-09-13T11:52:47.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:47.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94bbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:52:47.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94bbfff640 10 monclient: dump: 2024-09-13T11:52:47.355 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-09-13T11:52:47.356 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.356 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.356 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.356 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:47.356 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:47.356 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:47.356 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:47.356 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:47.356 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:47.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94bbfff640 10 monclient: _finish_auth 0 2024-09-13T11:52:47.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94bbfff640 10 monclient: _check_auth_tickets 2024-09-13T11:52:47.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:52:47.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94bbfff640 1 -- 172.21.15.122:0/1372285069 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94bc042070 con 0x7f94c4137630 2024-09-13T11:52:47.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c9da3640 5 monclient: authenticate success, global_id 4129 2024-09-13T11:52:47.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94bbfff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:52:47.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94bbfff640 1 -- 172.21.15.122:0/1372285069 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc007bf0 con 0x7f94c4137630 2024-09-13T11:52:47.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c9da3640 10 monclient: _renew_subs 2024-09-13T11:52:47.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c9da3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:52:47.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c9da3640 1 -- 172.21.15.122:0/1372285069 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94c4137330 con 0x7f94c4137630 2024-09-13T11:52:47.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:47.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94bbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:52:47.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94bbfff640 10 monclient: dump: 2024-09-13T11:52:47.357 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-09-13T11:52:47.357 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.357 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.357 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.357 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:47.358 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:47.358 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:47.358 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:47.358 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:47.358 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:47.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c1ffb640 4 set_mon_vals no callback set 2024-09-13T11:52:47.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c9da3640 10 monclient: _renew_subs 2024-09-13T11:52:47.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c9da3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:52:47.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94c9da3640 1 -- 172.21.15.122:0/1372285069 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94c41277b0 con 0x7f94c4137630 2024-09-13T11:52:47.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94bbfff640 1 -- 172.21.15.122:0/1372285069 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f94bc047460 con 0x7f94c4137630 2024-09-13T11:52:47.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.337+0000 7f94bbfff640 1 -- 172.21.15.122:0/1372285069 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f94bc007d90 con 0x7f94c4137630 2024-09-13T11:52:47.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.338+0000 7f94c9da3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:47.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.338+0000 7f94c9da3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:47.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.338+0000 7f94c9da3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:52:47.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.338+0000 7f94c9da3640 1 -- 172.21.15.122:0/1372285069 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94c406ff30 con 0x7f94c4137630 2024-09-13T11:52:47.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.340+0000 7f94bbfff640 1 -- 172.21.15.122:0/1372285069 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f94c406ff30 con 0x7f94c4137630 2024-09-13T11:52:47.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.340+0000 7f94bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:47.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.340+0000 7f94bbfff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:52:47.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.410+0000 7f94c9da3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "eca4429b-38cd-488a-b31d-c5856d701bde", "id": 3}] 2024-09-13T11:52:47.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.410+0000 7f94c9da3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "eca4429b-38cd-488a-b31d-c5856d701bde", "id": 3}] 2024-09-13T11:52:47.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.410+0000 7f94c9da3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:52:47.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.410+0000 7f94c9da3640 1 -- 172.21.15.122:0/1372285069 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd new", "uuid": "eca4429b-38cd-488a-b31d-c5856d701bde", "id": 3} v 0) -- 0x7f94c4138410 con 0x7f94c4137630 2024-09-13T11:52:47.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.416+0000 7f94bbfff640 1 -- 172.21.15.122:0/1372285069 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "eca4429b-38cd-488a-b31d-c5856d701bde", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f94bc0478e0 con 0x7f94c4137630 2024-09-13T11:52:47.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.416+0000 7f94bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "eca4429b-38cd-488a-b31d-c5856d701bde", "id": 3}] 2024-09-13T11:52:47.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.416+0000 7f94bbfff640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:52:47.419 INFO:teuthology.orchestra.run.smithi122.stdout:3 2024-09-13T11:52:47.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.417+0000 7f94c9da3640 10 monclient: shutdown 2024-09-13T11:52:47.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.417+0000 7f94c9da3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:52:47.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.417+0000 7f94c9da3640 1 -- 172.21.15.122:0/1372285069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94c4137630 msgr2=0x7f94c4123f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:47.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.417+0000 7f94c9da3640 1 --2- 172.21.15.122:0/1372285069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94c4137630 0x7f94c4123f50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009f40 tx=0x7f94bc004a40 comp rx=0 tx=0).stop 2024-09-13T11:52:47.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.417+0000 7f94c9da3640 1 -- 172.21.15.122:0/1372285069 shutdown_connections 2024-09-13T11:52:47.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.417+0000 7f94c9da3640 1 --2- 172.21.15.122:0/1372285069 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94c406ee30 0x7f94c41cd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.417+0000 7f94c9da3640 1 --2- 172.21.15.122:0/1372285069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94c4137630 0x7f94c4123f50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.417+0000 7f94c9da3640 1 --2- 172.21.15.122:0/1372285069 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94c406ea50 0x7f94c41ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.417+0000 7f94c9da3640 1 -- 172.21.15.122:0/1372285069 >> 172.21.15.122:0/1372285069 conn(0x7f94c40a6970 msgr2=0x7f94c409da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:52:47.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.417+0000 7f94c9da3640 1 -- 172.21.15.122:0/1372285069 shutdown_connections 2024-09-13T11:52:47.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.417+0000 7f94c9da3640 1 -- 172.21.15.122:0/1372285069 wait complete. 2024-09-13T11:52:47.429 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new b7ddc8f1-38d5-496a-9bf7-181eab7e5516 4 2024-09-13T11:52:47.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.524+0000 7f024ce40640 10 monclient: get_monmap_and_config 2024-09-13T11:52:47.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.524+0000 7f024ce40640 10 monclient: build_initial_monmap 2024-09-13T11:52:47.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.524+0000 7f024ce40640 10 monclient: monmap: 2024-09-13T11:52:47.527 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 0 2024-09-13T11:52:47.527 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.527 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:47.526190+0000 2024-09-13T11:52:47.527 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:47.526190+0000 2024-09-13T11:52:47.527 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 0 (unknown) 2024-09-13T11:52:47.528 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:47.528 INFO:teuthology.orchestra.run.smithi122.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:52:47.528 INFO:teuthology.orchestra.run.smithi122.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:52:47.528 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:52:47.528 INFO:teuthology.orchestra.run.smithi122.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:52:47.528 INFO:teuthology.orchestra.run.smithi122.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:52:47.528 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:47.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.525+0000 7f024ce40640 1 Processor -- start 2024-09-13T11:52:47.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.525+0000 7f024ce40640 1 -- start start 2024-09-13T11:52:47.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.525+0000 7f024ce40640 10 monclient: init 2024-09-13T11:52:47.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024ce40640 10 monclient: _reopen_session rank -1 2024-09-13T11:52:47.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024ce40640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-13T11:52:47.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024ce40640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f024806ea50 0x7f02480a2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024ce40640 10 monclient(hunting): picked mon.noname-c con 0x7f024806ea50 addr v2:172.21.15.122:3300/0 2024-09-13T11:52:47.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024ce40640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f024806ee30 0x7f024806f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024ce40640 10 monclient(hunting): picked mon.noname-a con 0x7f024806ee30 addr v2:172.21.15.100:3300/0 2024-09-13T11:52:47.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024ce40640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0248137630 addr v1:172.21.15.122:6789/0 2024-09-13T11:52:47.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024ce40640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02480a6440 con 0x7f0248137630 2024-09-13T11:52:47.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024ce40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024ce40640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f02480a1e50 con 0x7f024806ee30 2024-09-13T11:52:47.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024ce40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024ce40640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f02480a6680 con 0x7f024806ea50 2024-09-13T11:52:47.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024ce40640 10 monclient(hunting): _renew_subs 2024-09-13T11:52:47.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024ce40640 10 monclient(hunting): authenticate will time out at 747.839120s 2024-09-13T11:52:47.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024659c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f024806ea50 0x7f02480a2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f0246d9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f0248137630 0x7f0248137aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:56276/0 (socket says 172.21.15.122:56276) 2024-09-13T11:52:47.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f0246d9d640 1 -- 172.21.15.122:0/3487091645 learned_addr learned my addr 172.21.15.122:0/3487091645 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-09-13T11:52:47.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024659c640 10 monclient(hunting): get_auth_request con 0x7f024806ea50 auth_method 0 2024-09-13T11:52:47.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:47.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024659c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:47.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:47.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:47.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:47.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:47.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f0245d9b640 1 --2- 172.21.15.122:0/3487091645 >> v2:172.21.15.100:3300/0 conn(0x7f024806ee30 0x7f024806f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f0245d9b640 10 monclient(hunting): get_auth_request con 0x7f024806ee30 auth_method 0 2024-09-13T11:52:47.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f0245d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:47.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f0245d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:47.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f0245d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:47.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024659c640 10 monclient(hunting): handle_auth_done global_id 4132 payload 274 2024-09-13T11:52:47.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024659c640 1 -- 172.21.15.122:0/3487091645 >> v2:172.21.15.100:3300/0 conn(0x7f024806ee30 msgr2=0x7f024806f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:52:47.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024659c640 1 --2- 172.21.15.122:0/3487091645 >> v2:172.21.15.100:3300/0 conn(0x7f024806ee30 0x7f024806f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024659c640 1 -- 172.21.15.122:0/3487091645 >> v1:172.21.15.122:6789/0 conn(0x7f0248137630 legacy=0x7f0248137aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:47.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024659c640 10 monclient: _finish_hunting 0 2024-09-13T11:52:47.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024659c640 1 monclient: found mon.noname-c 2024-09-13T11:52:47.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:52:47.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024559a640 1 -- 172.21.15.122:0/3487091645 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1079660360 0 0) 0x7f02480a6440 con 0x7f0248137630 2024-09-13T11:52:47.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.122:3300/0 2024-09-13T11:52:47.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024659c640 1 -- 172.21.15.122:0/3487091645 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f024806fc60 con 0x7f024806ea50 2024-09-13T11:52:47.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f024559a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:52:47.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.526+0000 7f0245d9b640 1 --2- 172.21.15.122:0/3487091645 >> v2:172.21.15.100:3300/0 conn(0x7f024806ee30 0x7f024806f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:52:47.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.527+0000 7f024659c640 1 --2- 172.21.15.122:0/3487091645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f024806ea50 0x7f02480a2830 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f02480aaf30 tx=0x7f023802a080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5adb4e54fc7b875d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:52:47.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.527+0000 7f024559a640 1 -- 172.21.15.122:0/3487091645 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0238037070 con 0x7f024806ea50 2024-09-13T11:52:47.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.527+0000 7f024559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:47.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.527+0000 7f024559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:52:47.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.527+0000 7f024559a640 10 monclient: dump: 2024-09-13T11:52:47.538 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-09-13T11:52:47.539 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.539 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.539 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.539 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:47.539 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:47.539 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:47.539 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:47.539 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:47.539 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:47.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.527+0000 7f024559a640 10 monclient: _finish_auth 0 2024-09-13T11:52:47.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.527+0000 7f024559a640 10 monclient: _check_auth_tickets 2024-09-13T11:52:47.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.527+0000 7f024559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:52:47.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.527+0000 7f024559a640 1 -- 172.21.15.122:0/3487091645 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f023802add0 con 0x7f024806ea50 2024-09-13T11:52:47.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.527+0000 7f024ce40640 5 monclient: authenticate success, global_id 4132 2024-09-13T11:52:47.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.527+0000 7f024ce40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:52:47.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.527+0000 7f024559a640 10 monclient: handle_config config(0 keys) 2024-09-13T11:52:47.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.527+0000 7f024ce40640 10 monclient: get_monmap_and_config success 2024-09-13T11:52:47.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.527+0000 7f024559a640 1 -- 172.21.15.122:0/3487091645 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0238032660 con 0x7f024806ea50 2024-09-13T11:52:47.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.527+0000 7f024559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:47.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.527+0000 7f024ce40640 4 set_mon_vals no callback set 2024-09-13T11:52:47.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.527+0000 7f024559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:52:47.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.527+0000 7f024559a640 10 monclient: dump: 2024-09-13T11:52:47.540 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-09-13T11:52:47.540 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.540 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.540 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.540 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:47.541 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:47.541 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:47.541 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:47.541 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:47.541 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:47.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.527+0000 7f024ce40640 10 monclient: shutdown 2024-09-13T11:52:47.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.527+0000 7f024ce40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:52:47.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.527+0000 7f024ce40640 1 -- 172.21.15.122:0/3487091645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f024806ea50 msgr2=0x7f02480a2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:47.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.527+0000 7f024ce40640 1 --2- 172.21.15.122:0/3487091645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f024806ea50 0x7f02480a2830 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f02480aaf30 tx=0x7f023802a080 comp rx=0 tx=0).stop 2024-09-13T11:52:47.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.528+0000 7f024ce40640 1 -- 172.21.15.122:0/3487091645 shutdown_connections 2024-09-13T11:52:47.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.528+0000 7f024ce40640 1 --2- 172.21.15.122:0/3487091645 >> v2:172.21.15.100:3300/0 conn(0x7f024806ee30 0x7f024806f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.528+0000 7f024ce40640 1 --2- 172.21.15.122:0/3487091645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f024806ea50 0x7f02480a2830 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.528+0000 7f024ce40640 1 -- 172.21.15.122:0/3487091645 >> 172.21.15.122:0/3487091645 conn(0x7f02480a6970 msgr2=0x7f02480a6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:52:47.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.528+0000 7f024ce40640 1 -- 172.21.15.122:0/3487091645 shutdown_connections 2024-09-13T11:52:47.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.528+0000 7f024ce40640 1 -- 172.21.15.122:0/3487091645 wait complete. 2024-09-13T11:52:47.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.528+0000 7f024ce40640 10 monclient: build_initial_monmap 2024-09-13T11:52:47.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.528+0000 7f024ce40640 10 monclient: monmap: 2024-09-13T11:52:47.542 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 0 2024-09-13T11:52:47.542 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:52:47.542 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 0.000000 2024-09-13T11:52:47.542 INFO:teuthology.orchestra.run.smithi122.stderr:created 0.000000 2024-09-13T11:52:47.542 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 0 (unknown) 2024-09-13T11:52:47.543 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:47.543 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:52:47.543 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:52:47.543 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:52:47.543 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:47.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f024ce40640 1 Processor -- start 2024-09-13T11:52:47.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f024ce40640 1 -- start start 2024-09-13T11:52:47.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f024ce40640 10 monclient: init 2024-09-13T11:52:47.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f024ce40640 10 monclient: _reopen_session rank -1 2024-09-13T11:52:47.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f024ce40640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:52:47.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f024ce40640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f024806ea50 0x7f0248123cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f024ce40640 10 monclient(hunting): picked mon.noname-c con 0x7f024806ea50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:52:47.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f024ce40640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f024806ee30 0x7f0248124200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f024ce40640 10 monclient(hunting): picked mon.noname-b con 0x7f024806ee30 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:52:47.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f024ce40640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0248137630 0x7f0248124740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f024ce40640 10 monclient(hunting): picked mon.noname-a con 0x7f0248137630 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:52:47.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f024ce40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f024ce40640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f02480a5c70 con 0x7f0248137630 2024-09-13T11:52:47.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f024ce40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f024ce40640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f02480a5af0 con 0x7f024806ee30 2024-09-13T11:52:47.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f024ce40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f024ce40640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f02481381b0 con 0x7f024806ea50 2024-09-13T11:52:47.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f024ce40640 10 monclient(hunting): _renew_subs 2024-09-13T11:52:47.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f024ce40640 10 monclient(hunting): authenticate will time out at 747.842602s 2024-09-13T11:52:47.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f0245d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f024806ee30 0x7f0248124200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f0245d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f024806ee30 0x7f0248124200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:50132/0 (socket says 172.21.15.122:50132) 2024-09-13T11:52:47.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f0245d9b640 1 -- 172.21.15.122:0/3028384662 learned_addr learned my addr 172.21.15.122:0/3028384662 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-09-13T11:52:47.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f0245d9b640 10 monclient(hunting): get_auth_request con 0x7f024806ee30 auth_method 0 2024-09-13T11:52:47.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f0245d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:47.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f0245d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:47.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f0245d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:47.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f0245d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:47.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f0245d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:47.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f0245d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:47.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f024659c640 1 --2- 172.21.15.122:0/3028384662 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f024806ea50 0x7f0248123cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f0245d9b640 10 monclient(hunting): handle_auth_done global_id 4138 payload 824 2024-09-13T11:52:47.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f0246d9d640 1 --2- 172.21.15.122:0/3028384662 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0248137630 0x7f0248124740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f0245d9b640 1 -- 172.21.15.122:0/3028384662 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f024806ea50 msgr2=0x7f0248123cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:47.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f0245d9b640 1 --2- 172.21.15.122:0/3028384662 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f024806ea50 0x7f0248123cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.529+0000 7f0245d9b640 1 -- 172.21.15.122:0/3028384662 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0248137630 msgr2=0x7f0248124740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:47.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f0245d9b640 1 --2- 172.21.15.122:0/3028384662 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0248137630 0x7f0248124740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f0245d9b640 10 monclient: _finish_hunting 0 2024-09-13T11:52:47.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f0245d9b640 1 monclient: found mon.noname-b 2024-09-13T11:52:47.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f0245d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:52:47.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f0245d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.122:3300/0 2024-09-13T11:52:47.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f0245d9b640 1 -- 172.21.15.122:0/3028384662 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0248127800 con 0x7f024806ee30 2024-09-13T11:52:47.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f024659c640 10 monclient: get_auth_request con 0x7f024806ea50 auth_method 0 2024-09-13T11:52:47.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f0246d9d640 10 monclient: get_auth_request con 0x7f0248137630 auth_method 0 2024-09-13T11:52:47.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f024659c640 1 --2- 172.21.15.122:0/3028384662 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f024806ea50 0x7f0248123cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:52:47.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f0246d9d640 1 --2- 172.21.15.122:0/3028384662 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0248137630 0x7f0248124740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:52:47.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f0245d9b640 1 --2- 172.21.15.122:0/3028384662 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f024806ee30 0x7f0248124200 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f023000e7e0 tx=0x7f023000eca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:52:47.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f023effd640 1 -- 172.21.15.122:0/3028384662 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023000f040 con 0x7f024806ee30 2024-09-13T11:52:47.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f023effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:47.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f023effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:52:47.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f023effd640 10 monclient: dump: 2024-09-13T11:52:47.547 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-09-13T11:52:47.547 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.547 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.547 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.547 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:47.547 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:47.547 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:47.547 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:47.547 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:47.547 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:47.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f023effd640 10 monclient: _finish_auth 0 2024-09-13T11:52:47.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f023effd640 10 monclient: _check_auth_tickets 2024-09-13T11:52:47.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f023effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:52:47.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f023effd640 1 -- 172.21.15.122:0/3028384662 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0230004030 con 0x7f024806ee30 2024-09-13T11:52:47.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f024ce40640 5 monclient: authenticate success, global_id 4138 2024-09-13T11:52:47.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f024ce40640 10 monclient: _renew_subs 2024-09-13T11:52:47.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f024ce40640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:52:47.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f024ce40640 1 -- 172.21.15.122:0/3028384662 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0248127ae0 con 0x7f024806ee30 2024-09-13T11:52:47.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f023effd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:52:47.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f023effd640 1 -- 172.21.15.122:0/3028384662 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0230004400 con 0x7f024806ee30 2024-09-13T11:52:47.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f023effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:47.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f023effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:52:47.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f023effd640 10 monclient: dump: 2024-09-13T11:52:47.548 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-09-13T11:52:47.548 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.549 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.549 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.549 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:47.549 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:47.549 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:47.549 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:47.549 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:47.549 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:47.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f0244d99640 4 set_mon_vals no callback set 2024-09-13T11:52:47.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f024ce40640 10 monclient: _renew_subs 2024-09-13T11:52:47.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f024ce40640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:52:47.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f024ce40640 1 -- 172.21.15.122:0/3028384662 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02480a10d0 con 0x7f024806ee30 2024-09-13T11:52:47.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f023effd640 1 -- 172.21.15.122:0/3028384662 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f023000ca00 con 0x7f024806ee30 2024-09-13T11:52:47.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.530+0000 7f023effd640 1 -- 172.21.15.122:0/3028384662 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f0230013a60 con 0x7f024806ee30 2024-09-13T11:52:47.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.531+0000 7f024ce40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:47.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.531+0000 7f024ce40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:47.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.531+0000 7f024ce40640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:52:47.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.531+0000 7f024ce40640 1 -- 172.21.15.122:0/3028384662 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02480a2830 con 0x7f024806ee30 2024-09-13T11:52:47.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.533+0000 7f023effd640 1 -- 172.21.15.122:0/3028384662 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f0230017020 con 0x7f024806ee30 2024-09-13T11:52:47.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.533+0000 7f023effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:47.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.533+0000 7f023effd640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:52:47.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.603+0000 7f024ce40640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b7ddc8f1-38d5-496a-9bf7-181eab7e5516", "id": 4}] 2024-09-13T11:52:47.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.603+0000 7f024ce40640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b7ddc8f1-38d5-496a-9bf7-181eab7e5516", "id": 4}] 2024-09-13T11:52:47.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.603+0000 7f024ce40640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:52:47.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.603+0000 7f024ce40640 1 -- 172.21.15.122:0/3028384662 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b7ddc8f1-38d5-496a-9bf7-181eab7e5516", "id": 4} v 0) -- 0x7f0248069100 con 0x7f024806ee30 2024-09-13T11:52:47.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.608+0000 7f023effd640 1 -- 172.21.15.122:0/3028384662 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b7ddc8f1-38d5-496a-9bf7-181eab7e5516", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f0230013cd0 con 0x7f024806ee30 2024-09-13T11:52:47.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.608+0000 7f023effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b7ddc8f1-38d5-496a-9bf7-181eab7e5516", "id": 4}] 2024-09-13T11:52:47.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.608+0000 7f023effd640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:52:47.611 INFO:teuthology.orchestra.run.smithi122.stdout:4 2024-09-13T11:52:47.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.609+0000 7f024ce40640 10 monclient: shutdown 2024-09-13T11:52:47.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.609+0000 7f024ce40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:52:47.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.609+0000 7f024ce40640 1 -- 172.21.15.122:0/3028384662 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f024806ee30 msgr2=0x7f0248124200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:47.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.609+0000 7f024ce40640 1 --2- 172.21.15.122:0/3028384662 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f024806ee30 0x7f0248124200 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f023000e7e0 tx=0x7f023000eca0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.610+0000 7f024ce40640 1 -- 172.21.15.122:0/3028384662 shutdown_connections 2024-09-13T11:52:47.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.610+0000 7f024ce40640 1 --2- 172.21.15.122:0/3028384662 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0248137630 0x7f0248124740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.610+0000 7f024ce40640 1 --2- 172.21.15.122:0/3028384662 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f024806ee30 0x7f0248124200 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.610+0000 7f024ce40640 1 --2- 172.21.15.122:0/3028384662 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f024806ea50 0x7f0248123cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.610+0000 7f024ce40640 1 -- 172.21.15.122:0/3028384662 >> 172.21.15.122:0/3028384662 conn(0x7f02480a6970 msgr2=0x7f02480a9620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:52:47.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.610+0000 7f024ce40640 1 -- 172.21.15.122:0/3028384662 shutdown_connections 2024-09-13T11:52:47.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.610+0000 7f024ce40640 1 -- 172.21.15.122:0/3028384662 wait complete. 2024-09-13T11:52:47.622 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new 9010f8e0-c97b-451c-8b29-3ccb99e982a3 5 2024-09-13T11:52:47.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.718+0000 7fd47f844640 10 monclient: get_monmap_and_config 2024-09-13T11:52:47.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.718+0000 7fd47f844640 10 monclient: build_initial_monmap 2024-09-13T11:52:47.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.718+0000 7fd47f844640 10 monclient: monmap: 2024-09-13T11:52:47.721 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 0 2024-09-13T11:52:47.721 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.721 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:47.719721+0000 2024-09-13T11:52:47.721 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:47.719721+0000 2024-09-13T11:52:47.721 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 0 (unknown) 2024-09-13T11:52:47.721 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:47.721 INFO:teuthology.orchestra.run.smithi122.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:52:47.721 INFO:teuthology.orchestra.run.smithi122.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:52:47.721 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:52:47.721 INFO:teuthology.orchestra.run.smithi122.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:52:47.721 INFO:teuthology.orchestra.run.smithi122.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:52:47.721 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:47.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.718+0000 7fd47f844640 1 Processor -- start 2024-09-13T11:52:47.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47f844640 1 -- start start 2024-09-13T11:52:47.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47f844640 10 monclient: init 2024-09-13T11:52:47.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47f844640 10 monclient: _reopen_session rank -1 2024-09-13T11:52:47.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47f844640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-13T11:52:47.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47f844640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd47806ea50 0x7fd4780a2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47f844640 10 monclient(hunting): picked mon.noname-b con 0x7fd47806ea50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:52:47.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47f844640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd47806ee30 0x7fd47806f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47f844640 10 monclient(hunting): picked mon.noname-a con 0x7fd47806ee30 addr v2:172.21.15.100:3300/0 2024-09-13T11:52:47.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47f844640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd478137630 0x7fd478137aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47f844640 10 monclient(hunting): picked mon.noname-c con 0x7fd478137630 addr v2:172.21.15.122:3300/0 2024-09-13T11:52:47.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47f844640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47f844640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7fd4780a1e50 con 0x7fd47806ee30 2024-09-13T11:52:47.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47f844640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47f844640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7fd4780a6440 con 0x7fd478137630 2024-09-13T11:52:47.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47f844640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47f844640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fd4780a6680 con 0x7fd47806ea50 2024-09-13T11:52:47.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47f844640 10 monclient(hunting): _renew_subs 2024-09-13T11:52:47.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47f844640 10 monclient(hunting): authenticate will time out at 748.032608s 2024-09-13T11:52:47.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47dde1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd478137630 0x7fd478137aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47dde1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd478137630 0x7fd478137aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:50134/0 (socket says 172.21.15.122:50134) 2024-09-13T11:52:47.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47dde1640 1 -- 172.21.15.122:0/2073194812 learned_addr learned my addr 172.21.15.122:0/2073194812 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-09-13T11:52:47.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47dde1640 10 monclient(hunting): get_auth_request con 0x7fd478137630 auth_method 0 2024-09-13T11:52:47.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47dde1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:47.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47dde1640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:47.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47dde1640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:47.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47dde1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:47.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47dde1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:47.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47dde1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:47.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47d5e0640 1 --2- 172.21.15.122:0/2073194812 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd47806ea50 0x7fd4780a2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47cddf640 1 --2- 172.21.15.122:0/2073194812 >> v2:172.21.15.100:3300/0 conn(0x7fd47806ee30 0x7fd47806f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47d5e0640 10 monclient(hunting): get_auth_request con 0x7fd47806ea50 auth_method 0 2024-09-13T11:52:47.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47d5e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:47.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47d5e0640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:47.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47d5e0640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:47.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47cddf640 10 monclient(hunting): get_auth_request con 0x7fd47806ee30 auth_method 0 2024-09-13T11:52:47.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47cddf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:47.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47cddf640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:47.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.719+0000 7fd47cddf640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:47.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd47dde1640 10 monclient(hunting): handle_auth_done global_id 4141 payload 274 2024-09-13T11:52:47.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd47dde1640 1 -- 172.21.15.122:0/2073194812 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd47806ea50 msgr2=0x7fd4780a2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:52:47.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd47dde1640 1 --2- 172.21.15.122:0/2073194812 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd47806ea50 0x7fd4780a2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd47dde1640 1 -- 172.21.15.122:0/2073194812 >> v2:172.21.15.100:3300/0 conn(0x7fd47806ee30 msgr2=0x7fd47806f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:52:47.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd47dde1640 1 --2- 172.21.15.122:0/2073194812 >> v2:172.21.15.100:3300/0 conn(0x7fd47806ee30 0x7fd47806f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd47dde1640 10 monclient: _finish_hunting 0 2024-09-13T11:52:47.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd47dde1640 1 monclient: found mon.noname-c 2024-09-13T11:52:47.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd47dde1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:52:47.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd47dde1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.122:3300/0 2024-09-13T11:52:47.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd47dde1640 1 -- 172.21.15.122:0/2073194812 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd47806fc60 con 0x7fd478137630 2024-09-13T11:52:47.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd47d5e0640 1 --2- 172.21.15.122:0/2073194812 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd47806ea50 0x7fd4780a2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:52:47.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd47cddf640 1 --2- 172.21.15.122:0/2073194812 >> v2:172.21.15.100:3300/0 conn(0x7fd47806ee30 0x7fd47806f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:52:47.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd47dde1640 1 --2- 172.21.15.122:0/2073194812 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd478137630 0x7fd478137aa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd47806fba0 tx=0x7fd47002a570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c49060b5333bed6f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:52:47.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd46ffff640 1 -- 172.21.15.122:0/2073194812 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd470036070 con 0x7fd478137630 2024-09-13T11:52:47.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd46ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:47.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd46ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:52:47.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd46ffff640 10 monclient: dump: 2024-09-13T11:52:47.732 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-09-13T11:52:47.733 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.733 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.733 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.733 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:47.733 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:47.733 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:47.733 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:47.733 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:47.733 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:47.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd46ffff640 10 monclient: _finish_auth 0 2024-09-13T11:52:47.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd46ffff640 10 monclient: _check_auth_tickets 2024-09-13T11:52:47.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd46ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:52:47.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd46ffff640 1 -- 172.21.15.122:0/2073194812 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd470031600 con 0x7fd478137630 2024-09-13T11:52:47.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd47f844640 5 monclient: authenticate success, global_id 4141 2024-09-13T11:52:47.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd47f844640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:52:47.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd46ffff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:52:47.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd46ffff640 1 -- 172.21.15.122:0/2073194812 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4700319d0 con 0x7fd478137630 2024-09-13T11:52:47.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd47f844640 10 monclient: get_monmap_and_config success 2024-09-13T11:52:47.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd47f844640 4 set_mon_vals no callback set 2024-09-13T11:52:47.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd46ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:47.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd46ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:52:47.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.720+0000 7fd46ffff640 10 monclient: dump: 2024-09-13T11:52:47.734 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-09-13T11:52:47.734 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.734 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.734 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.734 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:47.735 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:47.735 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:47.735 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:47.735 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:47.735 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:47.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.721+0000 7fd47f844640 10 monclient: shutdown 2024-09-13T11:52:47.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.721+0000 7fd47f844640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:52:47.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.721+0000 7fd47f844640 1 -- 172.21.15.122:0/2073194812 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd478137630 msgr2=0x7fd478137aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:47.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.721+0000 7fd47f844640 1 --2- 172.21.15.122:0/2073194812 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd478137630 0x7fd478137aa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd47806fba0 tx=0x7fd47002a570 comp rx=0 tx=0).stop 2024-09-13T11:52:47.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.721+0000 7fd47f844640 1 -- 172.21.15.122:0/2073194812 shutdown_connections 2024-09-13T11:52:47.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.721+0000 7fd47f844640 1 --2- 172.21.15.122:0/2073194812 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd478137630 0x7fd478137aa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.721+0000 7fd47f844640 1 --2- 172.21.15.122:0/2073194812 >> v2:172.21.15.100:3300/0 conn(0x7fd47806ee30 0x7fd47806f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.721+0000 7fd47f844640 1 --2- 172.21.15.122:0/2073194812 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd47806ea50 0x7fd4780a2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.721+0000 7fd47f844640 1 -- 172.21.15.122:0/2073194812 >> 172.21.15.122:0/2073194812 conn(0x7fd4780a6970 msgr2=0x7fd4780a6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:52:47.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.721+0000 7fd47f844640 1 -- 172.21.15.122:0/2073194812 shutdown_connections 2024-09-13T11:52:47.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.721+0000 7fd47f844640 1 -- 172.21.15.122:0/2073194812 wait complete. 2024-09-13T11:52:47.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.722+0000 7fd47f844640 10 monclient: build_initial_monmap 2024-09-13T11:52:47.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.722+0000 7fd47f844640 10 monclient: monmap: 2024-09-13T11:52:47.736 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 0 2024-09-13T11:52:47.736 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:52:47.736 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 0.000000 2024-09-13T11:52:47.736 INFO:teuthology.orchestra.run.smithi122.stderr:created 0.000000 2024-09-13T11:52:47.736 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 0 (unknown) 2024-09-13T11:52:47.736 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:47.736 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:52:47.736 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:52:47.736 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:52:47.737 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:47.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.722+0000 7fd47f844640 1 Processor -- start 2024-09-13T11:52:47.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.722+0000 7fd47f844640 1 -- start start 2024-09-13T11:52:47.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.722+0000 7fd47f844640 10 monclient: init 2024-09-13T11:52:47.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47f844640 10 monclient: _reopen_session rank -1 2024-09-13T11:52:47.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47f844640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:52:47.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47f844640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd47806ea50 0x7fd478123d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47f844640 10 monclient(hunting): picked mon.noname-b con 0x7fd47806ea50 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:52:47.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47f844640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd47806ee30 0x7fd4781242d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47f844640 10 monclient(hunting): picked mon.noname-c con 0x7fd47806ee30 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:52:47.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47f844640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd478137630 0x7fd478124810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47f844640 10 monclient(hunting): picked mon.noname-a con 0x7fd478137630 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:52:47.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47f844640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47f844640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fd4780a5c70 con 0x7fd478137630 2024-09-13T11:52:47.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47f844640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47f844640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fd4780a5af0 con 0x7fd47806ea50 2024-09-13T11:52:47.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47f844640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47f844640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fd478137fe0 con 0x7fd47806ee30 2024-09-13T11:52:47.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47f844640 10 monclient(hunting): _renew_subs 2024-09-13T11:52:47.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47f844640 10 monclient(hunting): authenticate will time out at 748.036263s 2024-09-13T11:52:47.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47d5e0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd47806ea50 0x7fd478123d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47d5e0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd47806ea50 0x7fd478123d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:50136/0 (socket says 172.21.15.122:50136) 2024-09-13T11:52:47.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47d5e0640 1 -- 172.21.15.122:0/726446729 learned_addr learned my addr 172.21.15.122:0/726446729 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-09-13T11:52:47.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47d5e0640 10 monclient(hunting): get_auth_request con 0x7fd47806ea50 auth_method 0 2024-09-13T11:52:47.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47d5e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:47.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47d5e0640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:47.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47d5e0640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:47.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47d5e0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:47.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47d5e0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:47.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47d5e0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:47.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47dde1640 1 --2- 172.21.15.122:0/726446729 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd478137630 0x7fd478124810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47cddf640 1 --2- 172.21.15.122:0/726446729 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd47806ee30 0x7fd4781242d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47d5e0640 10 monclient(hunting): handle_auth_done global_id 4144 payload 824 2024-09-13T11:52:47.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47d5e0640 1 -- 172.21.15.122:0/726446729 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd47806ee30 msgr2=0x7fd4781242d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:47.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47d5e0640 1 --2- 172.21.15.122:0/726446729 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd47806ee30 0x7fd4781242d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47d5e0640 1 -- 172.21.15.122:0/726446729 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd478137630 msgr2=0x7fd478124810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:47.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47d5e0640 1 --2- 172.21.15.122:0/726446729 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd478137630 0x7fd478124810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47d5e0640 10 monclient: _finish_hunting 0 2024-09-13T11:52:47.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47d5e0640 1 monclient: found mon.noname-b 2024-09-13T11:52:47.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47d5e0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:52:47.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47d5e0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.122:3300/0 2024-09-13T11:52:47.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47d5e0640 1 -- 172.21.15.122:0/726446729 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4780a10d0 con 0x7fd47806ea50 2024-09-13T11:52:47.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47dde1640 10 monclient: get_auth_request con 0x7fd478137630 auth_method 0 2024-09-13T11:52:47.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47dde1640 1 --2- 172.21.15.122:0/726446729 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd478137630 0x7fd478124810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:52:47.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47cddf640 10 monclient: get_auth_request con 0x7fd47806ee30 auth_method 0 2024-09-13T11:52:47.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47cddf640 1 --2- 172.21.15.122:0/726446729 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd47806ee30 0x7fd4781242d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:52:47.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47d5e0640 1 --2- 172.21.15.122:0/726446729 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd47806ea50 0x7fd478123d90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd47400e810 tx=0x7fd47400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:52:47.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd46dffb640 1 -- 172.21.15.122:0/726446729 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47400f040 con 0x7fd47806ea50 2024-09-13T11:52:47.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd46dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:47.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd46dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:52:47.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd46dffb640 10 monclient: dump: 2024-09-13T11:52:47.740 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-09-13T11:52:47.740 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.741 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.741 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.741 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:47.741 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:47.741 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:47.741 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:47.741 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:47.741 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:47.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd46dffb640 10 monclient: _finish_auth 0 2024-09-13T11:52:47.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd46dffb640 10 monclient: _check_auth_tickets 2024-09-13T11:52:47.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd46dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:52:47.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd46dffb640 1 -- 172.21.15.122:0/726446729 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd474004030 con 0x7fd47806ea50 2024-09-13T11:52:47.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47f844640 5 monclient: authenticate success, global_id 4144 2024-09-13T11:52:47.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47f844640 10 monclient: _renew_subs 2024-09-13T11:52:47.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47f844640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:52:47.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd47f844640 1 -- 172.21.15.122:0/726446729 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4781278d0 con 0x7fd47806ea50 2024-09-13T11:52:47.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd46dffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:52:47.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd46dffb640 1 -- 172.21.15.122:0/726446729 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd474004400 con 0x7fd47806ea50 2024-09-13T11:52:47.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd46dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:47.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd46dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:52:47.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd46dffb640 10 monclient: dump: 2024-09-13T11:52:47.742 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-09-13T11:52:47.742 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.742 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.742 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.742 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:47.742 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-09-13T11:52:47.742 INFO:teuthology.orchestra.run.smithi122.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:47.742 INFO:teuthology.orchestra.run.smithi122.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:47.743 INFO:teuthology.orchestra.run.smithi122.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:47.743 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:52:47.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.724+0000 7fd47f844640 10 monclient: _renew_subs 2024-09-13T11:52:47.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.723+0000 7fd46f7fe640 4 set_mon_vals no callback set 2024-09-13T11:52:47.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.724+0000 7fd47f844640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:52:47.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.724+0000 7fd47f844640 1 -- 172.21.15.122:0/726446729 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd478126d40 con 0x7fd47806ea50 2024-09-13T11:52:47.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.724+0000 7fd46dffb640 1 -- 172.21.15.122:0/726446729 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd47400ca60 con 0x7fd47806ea50 2024-09-13T11:52:47.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.724+0000 7fd46dffb640 1 -- 172.21.15.122:0/726446729 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fd474013990 con 0x7fd47806ea50 2024-09-13T11:52:47.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.725+0000 7fd47f844640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:47.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.725+0000 7fd47f844640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:47.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.725+0000 7fd47f844640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:52:47.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.725+0000 7fd47f844640 1 -- 172.21.15.122:0/726446729 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd478138410 con 0x7fd47806ea50 2024-09-13T11:52:47.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.726+0000 7fd46dffb640 1 -- 172.21.15.122:0/726446729 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fd4780a10d0 con 0x7fd47806ea50 2024-09-13T11:52:47.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.726+0000 7fd46dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:47.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.726+0000 7fd46dffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:52:47.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.798+0000 7fd47f844640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9010f8e0-c97b-451c-8b29-3ccb99e982a3", "id": 5}] 2024-09-13T11:52:47.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.798+0000 7fd47f844640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9010f8e0-c97b-451c-8b29-3ccb99e982a3", "id": 5}] 2024-09-13T11:52:47.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.798+0000 7fd47f844640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:52:47.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.798+0000 7fd47f844640 1 -- 172.21.15.122:0/726446729 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9010f8e0-c97b-451c-8b29-3ccb99e982a3", "id": 5} v 0) -- 0x7fd478127fd0 con 0x7fd47806ea50 2024-09-13T11:52:47.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.805+0000 7fd46dffb640 1 -- 172.21.15.122:0/726446729 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9010f8e0-c97b-451c-8b29-3ccb99e982a3", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fd474017020 con 0x7fd47806ea50 2024-09-13T11:52:47.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.805+0000 7fd46dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9010f8e0-c97b-451c-8b29-3ccb99e982a3", "id": 5}] 2024-09-13T11:52:47.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.805+0000 7fd46dffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:52:47.808 INFO:teuthology.orchestra.run.smithi122.stdout:5 2024-09-13T11:52:47.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.806+0000 7fd47f844640 10 monclient: shutdown 2024-09-13T11:52:47.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.806+0000 7fd47f844640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:52:47.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.806+0000 7fd47f844640 1 -- 172.21.15.122:0/726446729 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd47806ea50 msgr2=0x7fd478123d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:47.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.806+0000 7fd47f844640 1 --2- 172.21.15.122:0/726446729 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd47806ea50 0x7fd478123d90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd47400e810 tx=0x7fd47400ecd0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.806+0000 7fd47f844640 1 -- 172.21.15.122:0/726446729 shutdown_connections 2024-09-13T11:52:47.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.806+0000 7fd47f844640 1 --2- 172.21.15.122:0/726446729 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd478137630 0x7fd478124810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.806+0000 7fd47f844640 1 --2- 172.21.15.122:0/726446729 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd47806ee30 0x7fd4781242d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.806+0000 7fd47f844640 1 --2- 172.21.15.122:0/726446729 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd47806ea50 0x7fd478123d90 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.806+0000 7fd47f844640 1 -- 172.21.15.122:0/726446729 >> 172.21.15.122:0/726446729 conn(0x7fd4780a6970 msgr2=0x7fd4780a05b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:52:47.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.807+0000 7fd47f844640 1 -- 172.21.15.122:0/726446729 shutdown_connections 2024-09-13T11:52:47.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T11:52:47.807+0000 7fd47f844640 1 -- 172.21.15.122:0/726446729 wait complete. 2024-09-13T11:52:47.818 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-13T11:52:47.819 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-13T11:52:47.821 INFO:tasks.ceph.osd.0:Started 2024-09-13T11:52:47.822 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-13T11:52:47.822 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-13T11:52:47.824 INFO:tasks.ceph.osd.1:Started 2024-09-13T11:52:47.824 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-13T11:52:47.824 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-13T11:52:47.827 INFO:tasks.ceph.osd.2:Started 2024-09-13T11:52:47.827 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-13T11:52:47.827 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-13T11:52:47.829 INFO:tasks.ceph.osd.3:Started 2024-09-13T11:52:47.829 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-13T11:52:47.829 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-13T11:52:47.832 INFO:tasks.ceph.osd.4:Started 2024-09-13T11:52:47.832 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-13T11:52:47.832 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-13T11:52:47.834 INFO:tasks.ceph.osd.5:Started 2024-09-13T11:52:47.834 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T11:52:47.944 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.943+0000 7f0f5e5ec640 10 monclient: get_monmap_and_config 2024-09-13T11:52:47.945 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.943+0000 7f0f5e5ec640 10 monclient: build_initial_monmap 2024-09-13T11:52:47.945 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.943+0000 7f0f5e5ec640 10 monclient: monmap: 2024-09-13T11:52:47.945 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:52:47.945 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.945 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:47.944180+0000 2024-09-13T11:52:47.945 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:47.944180+0000 2024-09-13T11:52:47.945 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:52:47.945 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:47.945 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:52:47.945 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:52:47.945 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:52:47.945 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:52:47.946 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:52:47.946 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:52:47.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5e5ec640 1 Processor -- start 2024-09-13T11:52:47.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5e5ec640 1 -- start start 2024-09-13T11:52:47.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5e5ec640 10 monclient: init 2024-09-13T11:52:47.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5e5ec640 10 monclient: _reopen_session rank -1 2024-09-13T11:52:47.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5e5ec640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-13T11:52:47.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5e5ec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0f5806ea50 addr v1:172.21.15.100:6789/0 2024-09-13T11:52:47.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5e5ec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0f580a2860 addr v1:172.21.15.122:6789/0 2024-09-13T11:52:47.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5e5ec640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0f5813da00 0x7f0f5813fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5e5ec640 10 monclient(hunting): picked mon.noname-a con 0x7f0f5813da00 addr v2:172.21.15.100:3300/0 2024-09-13T11:52:47.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5e5ec640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f580a1590 con 0x7f0f5806ea50 2024-09-13T11:52:47.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5e5ec640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f580a1870 con 0x7f0f580a2860 2024-09-13T11:52:47.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5e5ec640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5e5ec640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f0f580a2cb0 con 0x7f0f5813da00 2024-09-13T11:52:47.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5e5ec640 10 monclient(hunting): _renew_subs 2024-09-13T11:52:47.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5e5ec640 10 monclient(hunting): authenticate will time out at 748.537659s 2024-09-13T11:52:47.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5cb89640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0f5813da00 0x7f0f5813fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.956 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5cb89640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0f5813da00 0x7f0f5813fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:37584/0 (socket says 172.21.15.100:37584) 2024-09-13T11:52:47.956 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5cb89640 1 -- 172.21.15.100:0/1446866422 learned_addr learned my addr 172.21.15.100:0/1446866422 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:52:47.956 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5cb89640 10 monclient(hunting): get_auth_request con 0x7f0f5813da00 auth_method 0 2024-09-13T11:52:47.956 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5cb89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:47.956 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5cb89640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:47.956 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5cb89640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:47.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5cb89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:47.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5cb89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:47.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.944+0000 7f0f5cb89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:47.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f5cb89640 10 monclient(hunting): handle_auth_done global_id 4152 payload 274 2024-09-13T11:52:47.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f5cb89640 1 -- 172.21.15.100:0/1446866422 >> v1:172.21.15.122:6789/0 conn(0x7f0f580a2860 legacy=0x7f0f5806f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:47.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f5cb89640 1 -- 172.21.15.100:0/1446866422 >> v1:172.21.15.100:6789/0 conn(0x7f0f5806ea50 legacy=0x7f0f5806ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:47.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f56ffd640 1 -- 172.21.15.100:0/1446866422 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1119568411 0 0) 0x7f0f580a1870 con 0x7f0f580a2860 2024-09-13T11:52:47.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f5cb89640 10 monclient: _finish_hunting 0 2024-09-13T11:52:47.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f5cb89640 1 monclient: found mon.noname-a 2024-09-13T11:52:47.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f5cb89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:52:47.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f5cb89640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:52:47.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f5cb89640 1 -- 172.21.15.100:0/1446866422 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f580a1b50 con 0x7f0f5813da00 2024-09-13T11:52:47.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f56ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:52:47.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f5cb89640 1 --2- 172.21.15.100:0/1446866422 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f5813da00 0x7f0f5813fdf0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0f50009e50 tx=0x7f0f5002a5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3b612e55e6492c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:52:47.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f56ffd640 1 -- 172.21.15.100:0/1446866422 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f50036070 con 0x7f0f5813da00 2024-09-13T11:52:47.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:47.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f56ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:52:47.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f56ffd640 10 monclient: dump: 2024-09-13T11:52:47.959 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:52:47.959 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.959 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.959 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.959 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:47.959 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:47.959 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:47.960 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:47.960 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:47.960 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:52:47.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f56ffd640 10 monclient: _finish_auth 0 2024-09-13T11:52:47.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f56ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:52:47.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:52:47.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f5e5ec640 5 monclient: authenticate success, global_id 4152 2024-09-13T11:52:47.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f56ffd640 1 -- 172.21.15.100:0/1446866422 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f50031600 con 0x7f0f5813da00 2024-09-13T11:52:47.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f5e5ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:52:47.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f56ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:52:47.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f5e5ec640 10 monclient: get_monmap_and_config success 2024-09-13T11:52:47.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f5e5ec640 4 set_mon_vals no callback set 2024-09-13T11:52:47.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f56ffd640 1 -- 172.21.15.100:0/1446866422 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f500319d0 con 0x7f0f5813da00 2024-09-13T11:52:47.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:47.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f56ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:52:47.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.945+0000 7f0f56ffd640 10 monclient: dump: 2024-09-13T11:52:47.962 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:52:47.962 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.962 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.962 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.962 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:47.962 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:47.962 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:47.962 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:47.962 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:47.963 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:52:47.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.946+0000 7f0f5e5ec640 10 monclient: shutdown 2024-09-13T11:52:47.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.946+0000 7f0f5e5ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:52:47.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.946+0000 7f0f5e5ec640 1 -- 172.21.15.100:0/1446866422 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f5813da00 msgr2=0x7f0f5813fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:47.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.946+0000 7f0f5e5ec640 1 --2- 172.21.15.100:0/1446866422 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f5813da00 0x7f0f5813fdf0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0f50009e50 tx=0x7f0f5002a5b0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.946+0000 7f0f5e5ec640 1 -- 172.21.15.100:0/1446866422 shutdown_connections 2024-09-13T11:52:47.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.946+0000 7f0f5e5ec640 1 --2- 172.21.15.100:0/1446866422 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f5813da00 0x7f0f5813fdf0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.946+0000 7f0f5e5ec640 1 -- 172.21.15.100:0/1446866422 >> 172.21.15.100:0/1446866422 conn(0x7f0f580a68e0 msgr2=0x7f0f580a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:52:47.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.946+0000 7f0f5e5ec640 1 -- 172.21.15.100:0/1446866422 shutdown_connections 2024-09-13T11:52:47.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.946+0000 7f0f5e5ec640 1 -- 172.21.15.100:0/1446866422 wait complete. 2024-09-13T11:52:47.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.947+0000 7f0f5e5ec640 10 monclient: build_initial_monmap 2024-09-13T11:52:47.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.947+0000 7f0f5e5ec640 10 monclient: monmap: 2024-09-13T11:52:47.964 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:52:47.964 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:52:47.964 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:52:47.964 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:52:47.964 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:52:47.964 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:47.964 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:52:47.964 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:52:47.964 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:52:47.964 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:52:47.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.947+0000 7f0f5e5ec640 1 Processor -- start 2024-09-13T11:52:47.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.947+0000 7f0f5e5ec640 1 -- start start 2024-09-13T11:52:47.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.947+0000 7f0f5e5ec640 10 monclient: init 2024-09-13T11:52:47.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.947+0000 7f0f5e5ec640 10 monclient: _reopen_session rank -1 2024-09-13T11:52:47.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.947+0000 7f0f5e5ec640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:52:47.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.947+0000 7f0f5e5ec640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f5806ea50 0x7f0f5809d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5e5ec640 10 monclient(hunting): picked mon.noname-a con 0x7f0f5806ea50 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:52:47.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5e5ec640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f580a2860 0x7f0f5809dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5e5ec640 10 monclient(hunting): picked mon.noname-c con 0x7f0f580a2860 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:52:47.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5e5ec640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f5813da00 0x7f0f5809e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:47.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5e5ec640 10 monclient(hunting): picked mon.noname-b con 0x7f0f5813da00 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:52:47.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5e5ec640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5e5ec640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f0f581404b0 con 0x7f0f5806ea50 2024-09-13T11:52:47.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5e5ec640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5e5ec640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f0f58140330 con 0x7f0f5813da00 2024-09-13T11:52:47.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5e5ec640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:47.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5e5ec640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f0f58140630 con 0x7f0f580a2860 2024-09-13T11:52:47.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5e5ec640 10 monclient(hunting): _renew_subs 2024-09-13T11:52:47.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5e5ec640 10 monclient(hunting): authenticate will time out at 748.541058s 2024-09-13T11:52:47.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f57fff640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f5806ea50 0x7f0f5809d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f577fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f580a2860 0x7f0f5809dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f57fff640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f5806ea50 0x7f0f5809d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:37592/0 (socket says 172.21.15.100:37592) 2024-09-13T11:52:47.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f57fff640 1 -- 172.21.15.100:0/766150004 learned_addr learned my addr 172.21.15.100:0/766150004 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:52:47.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f57fff640 10 monclient(hunting): get_auth_request con 0x7f0f5806ea50 auth_method 0 2024-09-13T11:52:47.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:47.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f57fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:47.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:47.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f577fe640 10 monclient(hunting): get_auth_request con 0x7f0f580a2860 auth_method 0 2024-09-13T11:52:47.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:47.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f577fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:47.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:47.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:47.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:47.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:47.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:47.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:47.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:47.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5cb89640 1 --2- 172.21.15.100:0/766150004 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f5813da00 0x7f0f5809e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:47.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5cb89640 10 monclient(hunting): get_auth_request con 0x7f0f5813da00 auth_method 0 2024-09-13T11:52:47.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5cb89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:47.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5cb89640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:47.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5cb89640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:47.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5cb89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:47.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5cb89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:47.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5cb89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:47.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f57fff640 10 monclient(hunting): handle_auth_done global_id 4158 payload 824 2024-09-13T11:52:47.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f57fff640 1 -- 172.21.15.100:0/766150004 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f580a2860 msgr2=0x7f0f5809dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:47.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f57fff640 1 --2- 172.21.15.100:0/766150004 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f580a2860 0x7f0f5809dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f57fff640 1 -- 172.21.15.100:0/766150004 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f5813da00 msgr2=0x7f0f5809e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:47.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f57fff640 1 --2- 172.21.15.100:0/766150004 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f5813da00 0x7f0f5809e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:47.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f57fff640 10 monclient: _finish_hunting 0 2024-09-13T11:52:47.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f57fff640 1 monclient: found mon.noname-a 2024-09-13T11:52:47.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:52:47.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f57fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:52:47.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f57fff640 1 -- 172.21.15.100:0/766150004 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f580a0720 con 0x7f0f5806ea50 2024-09-13T11:52:47.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f577fe640 1 --2- 172.21.15.100:0/766150004 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f580a2860 0x7f0f5809dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:52:47.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f57fff640 1 --2- 172.21.15.100:0/766150004 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f5806ea50 0x7f0f5809d770 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0f4800c730 tx=0x7f0f4800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:52:47.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f54ff9640 1 -- 172.21.15.100:0/766150004 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f48017070 con 0x7f0f5806ea50 2024-09-13T11:52:47.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f54ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:47.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f54ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:52:47.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f54ff9640 10 monclient: dump: 2024-09-13T11:52:47.969 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:52:47.969 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.969 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.969 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.969 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:47.969 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:47.969 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:47.970 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:47.970 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:47.970 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:52:47.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f54ff9640 10 monclient: _finish_auth 0 2024-09-13T11:52:47.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f54ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:52:47.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:52:47.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f54ff9640 1 -- 172.21.15.100:0/766150004 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f48005290 con 0x7f0f5806ea50 2024-09-13T11:52:47.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5e5ec640 5 monclient: authenticate success, global_id 4158 2024-09-13T11:52:47.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5e5ec640 10 monclient: _renew_subs 2024-09-13T11:52:47.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5e5ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:52:47.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5e5ec640 1 -- 172.21.15.100:0/766150004 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f58120d70 con 0x7f0f5806ea50 2024-09-13T11:52:47.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f54ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:52:47.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f54ff9640 1 -- 172.21.15.100:0/766150004 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f4800a880 con 0x7f0f5806ea50 2024-09-13T11:52:47.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f54ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:47.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f54ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:52:47.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f54ff9640 10 monclient: dump: 2024-09-13T11:52:47.971 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:52:47.971 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:47.971 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.971 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:47.971 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:47.971 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:47.971 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:47.971 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:47.971 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:47.971 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:52:47.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5e5ec640 10 monclient: _renew_subs 2024-09-13T11:52:47.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f567fc640 4 set_mon_vals no callback set 2024-09-13T11:52:47.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5e5ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:52:47.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.948+0000 7f0f5e5ec640 1 -- 172.21.15.100:0/766150004 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f58120a90 con 0x7f0f5806ea50 2024-09-13T11:52:47.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.949+0000 7f0f54ff9640 1 -- 172.21.15.100:0/766150004 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0f4801a070 con 0x7f0f5806ea50 2024-09-13T11:52:47.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.949+0000 7f0f54ff9640 1 -- 172.21.15.100:0/766150004 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f0f48003b40 con 0x7f0f5806ea50 2024-09-13T11:52:47.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.949+0000 7f0f5e5ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:47.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.949+0000 7f0f5e5ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:47.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.949+0000 7f0f5e5ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:52:47.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.949+0000 7f0f5e5ec640 1 -- 172.21.15.100:0/766150004 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f5806ee30 con 0x7f0f5806ea50 2024-09-13T11:52:47.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.951+0000 7f0f54ff9640 1 -- 172.21.15.100:0/766150004 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f0f48021020 con 0x7f0f5806ea50 2024-09-13T11:52:47.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.951+0000 7f0f54ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:47.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:47.951+0000 7f0f54ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:52:48.011 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:48.009+0000 7f0f5e5ec640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:52:48.011 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:48.009+0000 7f0f5e5ec640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:52:48.011 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:48.009+0000 7f0f5e5ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:52:48.011 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:48.009+0000 7f0f5e5ec640 1 -- 172.21.15.100:0/766150004 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0f580a0060 con 0x7f0f5806ea50 2024-09-13T11:52:48.012 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:48.009+0000 7f0f54ff9640 1 -- 172.21.15.100:0/766150004 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f0f480035f0 con 0x7f0f5806ea50 2024-09-13T11:52:48.012 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:48.009+0000 7f0f54ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:52:48.012 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:48.009+0000 7f0f54ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:52:48.012 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:52:48.013 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":8,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","created":"2024-09-13T11:52:46.158794+0000","modified":"2024-09-13T11:52:47.800207+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T11:52:47.800207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ac784459-dbff-42ed-941e-70c741392dcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8d8d39e7-e8f1-499a-a01b-e7ba49844eae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e0bd3e30-0ff3-4067-83d3-ef8d84b29097","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"eca4429b-38cd-488a-b31d-c5856d701bde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b7ddc8f1-38d5-496a-9bf7-181eab7e5516","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9010f8e0-c97b-451c-8b29-3ccb99e982a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:52:48.014 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:48.010+0000 7f0f5e5ec640 10 monclient: shutdown 2024-09-13T11:52:48.014 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:48.010+0000 7f0f5e5ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:52:48.014 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:48.010+0000 7f0f5e5ec640 1 -- 172.21.15.100:0/766150004 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f5806ea50 msgr2=0x7f0f5809d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:48.015 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:48.010+0000 7f0f5e5ec640 1 --2- 172.21.15.100:0/766150004 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f5806ea50 0x7f0f5809d770 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0f4800c730 tx=0x7f0f4800cbf0 comp rx=0 tx=0).stop 2024-09-13T11:52:48.015 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:48.010+0000 7f0f5e5ec640 1 -- 172.21.15.100:0/766150004 shutdown_connections 2024-09-13T11:52:48.015 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:48.010+0000 7f0f5e5ec640 1 --2- 172.21.15.100:0/766150004 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f5813da00 0x7f0f5809e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:48.015 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:48.010+0000 7f0f5e5ec640 1 --2- 172.21.15.100:0/766150004 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f580a2860 0x7f0f5809dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:48.015 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:48.010+0000 7f0f5e5ec640 1 --2- 172.21.15.100:0/766150004 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f5806ea50 0x7f0f5809d770 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:48.015 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:48.010+0000 7f0f5e5ec640 1 -- 172.21.15.100:0/766150004 >> 172.21.15.100:0/766150004 conn(0x7f0f580a68e0 msgr2=0x7f0f580a3650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:52:48.015 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:48.010+0000 7f0f5e5ec640 1 -- 172.21.15.100:0/766150004 shutdown_connections 2024-09-13T11:52:48.016 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:52:48.010+0000 7f0f5e5ec640 1 -- 172.21.15.100:0/766150004 wait complete. 2024-09-13T11:52:48.022 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-13T11:52:48.023 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-13T11:52:48.628 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T11:52:48.719 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.718+0000 7f22b5abe640 10 monclient: get_monmap_and_config 2024-09-13T11:52:48.719 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.718+0000 7f22b5abe640 10 monclient: build_initial_monmap 2024-09-13T11:52:48.719 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.718+0000 7f22b5abe640 10 monclient: monmap: 2024-09-13T11:52:48.719 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-09-13T11:52:48.720 INFO:teuthology.misc.health.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:48.720 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-09-13T11:52:48.718946+0000 2024-09-13T11:52:48.720 INFO:teuthology.misc.health.smithi100.stderr:created 2024-09-13T11:52:48.718946+0000 2024-09-13T11:52:48.720 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:52:48.720 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:48.720 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:52:48.720 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:52:48.720 INFO:teuthology.misc.health.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:52:48.720 INFO:teuthology.misc.health.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:52:48.720 INFO:teuthology.misc.health.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:52:48.720 INFO:teuthology.misc.health.smithi100.stderr: 2024-09-13T11:52:48.728 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.718+0000 7f22b5abe640 1 Processor -- start 2024-09-13T11:52:48.728 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.718+0000 7f22b5abe640 1 -- start start 2024-09-13T11:52:48.728 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.718+0000 7f22b5abe640 10 monclient: init 2024-09-13T11:52:48.728 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22b5abe640 10 monclient: _reopen_session rank -1 2024-09-13T11:52:48.728 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22b5abe640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-13T11:52:48.729 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22b5abe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f22b0137b50 addr v1:172.21.15.100:6789/0 2024-09-13T11:52:48.729 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22b5abe640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f22b00a58a0 0x7f22b00a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:48.729 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22b5abe640 10 monclient(hunting): picked mon.noname-a con 0x7f22b00a58a0 addr v2:172.21.15.100:3300/0 2024-09-13T11:52:48.729 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22b5abe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22b013cb20 0x7f22b013ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:48.729 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22b5abe640 10 monclient(hunting): picked mon.noname-b con 0x7f22b013cb20 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:52:48.729 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22b5abe640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22b00a5620 con 0x7f22b0137b50 2024-09-13T11:52:48.729 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22b5abe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:48.729 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22b5abe640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f22b00a5ce0 con 0x7f22b00a58a0 2024-09-13T11:52:48.729 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22b5abe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:48.729 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22b5abe640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f22b00a6460 con 0x7f22b013cb20 2024-09-13T11:52:48.729 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22b5abe640 10 monclient(hunting): _renew_subs 2024-09-13T11:52:48.729 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22b5abe640 10 monclient(hunting): authenticate will time out at 749.312426s 2024-09-13T11:52:48.729 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22aeffd640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f22b00a58a0 0x7f22b00a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:48.730 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22aeffd640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f22b00a58a0 0x7f22b00a0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:37596/0 (socket says 172.21.15.100:37596) 2024-09-13T11:52:48.730 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22aeffd640 1 -- 172.21.15.100:0/2689349061 learned_addr learned my addr 172.21.15.100:0/2689349061 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:52:48.730 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22aeffd640 10 monclient(hunting): get_auth_request con 0x7f22b00a58a0 auth_method 0 2024-09-13T11:52:48.730 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:48.730 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:48.730 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:48.730 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22affff640 1 --2- 172.21.15.100:0/2689349061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22b013cb20 0x7f22b013ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:48.730 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22affff640 10 monclient(hunting): get_auth_request con 0x7f22b013cb20 auth_method 0 2024-09-13T11:52:48.730 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:48.730 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22affff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:48.730 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:48.730 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:48.730 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:48.731 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:48.731 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:48.731 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:48.731 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:48.731 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22aeffd640 10 monclient(hunting): handle_auth_done global_id 4161 payload 274 2024-09-13T11:52:48.731 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22aeffd640 1 -- 172.21.15.100:0/2689349061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22b013cb20 msgr2=0x7f22b013ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:52:48.731 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22aeffd640 1 --2- 172.21.15.100:0/2689349061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22b013cb20 0x7f22b013ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:48.731 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22aeffd640 1 -- 172.21.15.100:0/2689349061 >> v1:172.21.15.100:6789/0 conn(0x7f22b0137b50 legacy=0x7f22b0137f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:48.731 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22aeffd640 10 monclient: _finish_hunting 0 2024-09-13T11:52:48.731 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22aeffd640 1 monclient: found mon.noname-a 2024-09-13T11:52:48.731 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:52:48.731 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22aeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:52:48.731 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22aeffd640 1 -- 172.21.15.100:0/2689349061 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b00a6940 con 0x7f22b00a58a0 2024-09-13T11:52:48.732 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.719+0000 7f22affff640 1 --2- 172.21.15.100:0/2689349061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22b013cb20 0x7f22b013ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:52:48.732 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.720+0000 7f22aeffd640 1 --2- 172.21.15.100:0/2689349061 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f22b00a58a0 0x7f22b00a0e00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f22a4009870 tx=0x7f22a402a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ffb85f294418ab5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:52:48.732 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.720+0000 7f22ae7fc640 1 -- 172.21.15.100:0/2689349061 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a4037070 con 0x7f22b00a58a0 2024-09-13T11:52:48.732 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.720+0000 7f22ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:48.732 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.720+0000 7f22ae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:52:48.732 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.720+0000 7f22ae7fc640 10 monclient: dump: 2024-09-13T11:52:48.732 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-09-13T11:52:48.732 INFO:teuthology.misc.health.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:48.732 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:48.732 INFO:teuthology.misc.health.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:48.732 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:48.732 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:48.732 INFO:teuthology.misc.health.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:48.732 INFO:teuthology.misc.health.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:48.733 INFO:teuthology.misc.health.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:48.733 INFO:teuthology.misc.health.smithi100.stderr: 2024-09-13T11:52:48.733 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.720+0000 7f22ae7fc640 10 monclient: _finish_auth 0 2024-09-13T11:52:48.733 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.720+0000 7f22ae7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:52:48.733 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.720+0000 7f22ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:52:48.733 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.720+0000 7f22ae7fc640 1 -- 172.21.15.100:0/2689349061 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22a4032440 con 0x7f22b00a58a0 2024-09-13T11:52:48.733 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.720+0000 7f22b5abe640 5 monclient: authenticate success, global_id 4161 2024-09-13T11:52:48.733 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.720+0000 7f22b5abe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:52:48.733 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.720+0000 7f22ae7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:52:48.733 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.720+0000 7f22ae7fc640 1 -- 172.21.15.100:0/2689349061 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a4032810 con 0x7f22b00a58a0 2024-09-13T11:52:48.733 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.720+0000 7f22ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:48.733 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.720+0000 7f22ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:52:48.734 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.720+0000 7f22ae7fc640 10 monclient: dump: 2024-09-13T11:52:48.734 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-09-13T11:52:48.734 INFO:teuthology.misc.health.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:48.734 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:48.734 INFO:teuthology.misc.health.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:48.734 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:48.734 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:48.734 INFO:teuthology.misc.health.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:48.734 INFO:teuthology.misc.health.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:48.734 INFO:teuthology.misc.health.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:48.734 INFO:teuthology.misc.health.smithi100.stderr: 2024-09-13T11:52:48.734 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.720+0000 7f22b5abe640 10 monclient: get_monmap_and_config success 2024-09-13T11:52:48.734 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.720+0000 7f22b5abe640 4 set_mon_vals no callback set 2024-09-13T11:52:48.734 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.721+0000 7f22b5abe640 10 monclient: shutdown 2024-09-13T11:52:48.735 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.721+0000 7f22b5abe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:52:48.735 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.721+0000 7f22b5abe640 1 -- 172.21.15.100:0/2689349061 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f22b00a58a0 msgr2=0x7f22b00a0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:48.735 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.721+0000 7f22b5abe640 1 --2- 172.21.15.100:0/2689349061 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f22b00a58a0 0x7f22b00a0e00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f22a4009870 tx=0x7f22a402a130 comp rx=0 tx=0).stop 2024-09-13T11:52:48.735 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.721+0000 7f22b5abe640 1 -- 172.21.15.100:0/2689349061 shutdown_connections 2024-09-13T11:52:48.735 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.721+0000 7f22b5abe640 1 --2- 172.21.15.100:0/2689349061 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f22b00a58a0 0x7f22b00a0e00 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:48.735 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.721+0000 7f22b5abe640 1 --2- 172.21.15.100:0/2689349061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22b013cb20 0x7f22b013ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:48.735 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.721+0000 7f22b5abe640 1 -- 172.21.15.100:0/2689349061 >> 172.21.15.100:0/2689349061 conn(0x7f22b00a7040 msgr2=0x7f22b00a7450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:52:48.735 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.721+0000 7f22b5abe640 1 -- 172.21.15.100:0/2689349061 shutdown_connections 2024-09-13T11:52:48.735 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.721+0000 7f22b5abe640 1 -- 172.21.15.100:0/2689349061 wait complete. 2024-09-13T11:52:48.735 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.721+0000 7f22b5abe640 10 monclient: build_initial_monmap 2024-09-13T11:52:48.735 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.721+0000 7f22b5abe640 10 monclient: monmap: 2024-09-13T11:52:48.735 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-09-13T11:52:48.735 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:52:48.736 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-09-13T11:52:48.736 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-09-13T11:52:48.736 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:52:48.736 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:48.736 INFO:teuthology.misc.health.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:52:48.736 INFO:teuthology.misc.health.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:52:48.736 INFO:teuthology.misc.health.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:52:48.736 INFO:teuthology.misc.health.smithi100.stderr: 2024-09-13T11:52:48.736 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22b5abe640 1 Processor -- start 2024-09-13T11:52:48.736 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22b5abe640 1 -- start start 2024-09-13T11:52:48.736 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22b5abe640 10 monclient: init 2024-09-13T11:52:48.736 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22b5abe640 10 monclient: _reopen_session rank -1 2024-09-13T11:52:48.736 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22b5abe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:52:48.737 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22b5abe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f22b00a58a0 0x7f22b01cc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:48.737 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22b5abe640 10 monclient(hunting): picked mon.noname-a con 0x7f22b00a58a0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:52:48.737 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22b5abe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22b0137b50 0x7f22b0136a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:48.737 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22b5abe640 10 monclient(hunting): picked mon.noname-b con 0x7f22b0137b50 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:52:48.737 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22b5abe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22b013cb20 0x7f22b0071300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:48.737 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22b5abe640 10 monclient(hunting): picked mon.noname-c con 0x7f22b013cb20 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:52:48.737 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22b5abe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:48.737 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22b5abe640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f22b013f450 con 0x7f22b00a58a0 2024-09-13T11:52:48.737 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22b5abe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:48.737 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22b5abe640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f22b0138640 con 0x7f22b0137b50 2024-09-13T11:52:48.737 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22b5abe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:48.737 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22b5abe640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f22b013f5d0 con 0x7f22b013cb20 2024-09-13T11:52:48.737 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22b5abe640 10 monclient(hunting): _renew_subs 2024-09-13T11:52:48.737 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22b5abe640 10 monclient(hunting): authenticate will time out at 749.315875s 2024-09-13T11:52:48.738 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22af7fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f22b00a58a0 0x7f22b01cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:48.738 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22af7fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f22b00a58a0 0x7f22b01cc750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:37608/0 (socket says 172.21.15.100:37608) 2024-09-13T11:52:48.738 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22af7fe640 1 -- 172.21.15.100:0/473744406 learned_addr learned my addr 172.21.15.100:0/473744406 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:52:48.738 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22af7fe640 10 monclient(hunting): get_auth_request con 0x7f22b00a58a0 auth_method 0 2024-09-13T11:52:48.738 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:48.738 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:48.738 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:48.738 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22affff640 1 --2- 172.21.15.100:0/473744406 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22b013cb20 0x7f22b0071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:48.738 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.722+0000 7f22affff640 10 monclient(hunting): get_auth_request con 0x7f22b013cb20 auth_method 0 2024-09-13T11:52:48.738 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:48.738 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22affff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:48.738 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:48.738 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:48.739 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:48.739 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:48.739 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:48.739 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:48.739 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:48.739 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22aeffd640 1 --2- 172.21.15.100:0/473744406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22b0137b50 0x7f22b0136a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:48.739 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22aeffd640 10 monclient(hunting): get_auth_request con 0x7f22b0137b50 auth_method 0 2024-09-13T11:52:48.739 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:48.739 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:48.739 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:48.739 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22affff640 10 monclient(hunting): handle_auth_done global_id 4145 payload 824 2024-09-13T11:52:48.739 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22affff640 1 -- 172.21.15.100:0/473744406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22b0137b50 msgr2=0x7f22b0136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:48.739 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22affff640 1 --2- 172.21.15.100:0/473744406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22b0137b50 0x7f22b0136a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:48.739 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22affff640 1 -- 172.21.15.100:0/473744406 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f22b00a58a0 msgr2=0x7f22b01cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:48.740 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22affff640 1 --2- 172.21.15.100:0/473744406 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f22b00a58a0 0x7f22b01cc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:48.740 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22affff640 10 monclient: _finish_hunting 0 2024-09-13T11:52:48.740 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22affff640 1 monclient: found mon.noname-c 2024-09-13T11:52:48.740 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:52:48.740 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22affff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:52:48.740 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22affff640 1 -- 172.21.15.100:0/473744406 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b0072dc0 con 0x7f22b013cb20 2024-09-13T11:52:48.740 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22af7fe640 1 --2- 172.21.15.100:0/473744406 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f22b00a58a0 0x7f22b01cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:52:48.740 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22affff640 1 --2- 172.21.15.100:0/473744406 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22b013cb20 0x7f22b0071300 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f22a800ef90 tx=0x7f22a800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:52:48.740 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f2297fff640 1 -- 172.21.15.100:0/473744406 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a8018030 con 0x7f22b013cb20 2024-09-13T11:52:48.740 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f2297fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:48.740 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f2297fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:52:48.740 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f2297fff640 10 monclient: dump: 2024-09-13T11:52:48.740 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-09-13T11:52:48.741 INFO:teuthology.misc.health.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:48.741 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:48.741 INFO:teuthology.misc.health.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:48.741 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:48.741 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:48.741 INFO:teuthology.misc.health.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:48.741 INFO:teuthology.misc.health.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:48.741 INFO:teuthology.misc.health.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:48.741 INFO:teuthology.misc.health.smithi100.stderr: 2024-09-13T11:52:48.741 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f2297fff640 10 monclient: _finish_auth 0 2024-09-13T11:52:48.741 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f2297fff640 10 monclient: _check_auth_tickets 2024-09-13T11:52:48.741 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f2297fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:52:48.741 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f2297fff640 1 -- 172.21.15.100:0/473744406 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22a800ccd0 con 0x7f22b013cb20 2024-09-13T11:52:48.741 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22b5abe640 5 monclient: authenticate success, global_id 4145 2024-09-13T11:52:48.742 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f2297fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:52:48.742 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f2297fff640 1 -- 172.21.15.100:0/473744406 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a8003bb0 con 0x7f22b013cb20 2024-09-13T11:52:48.742 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22b5abe640 10 monclient: _renew_subs 2024-09-13T11:52:48.742 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22b5abe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:52:48.742 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22b5abe640 1 -- 172.21.15.100:0/473744406 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22b013eee0 con 0x7f22b013cb20 2024-09-13T11:52:48.742 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f2297fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:48.742 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f2297fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:52:48.742 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f2297fff640 10 monclient: dump: 2024-09-13T11:52:48.742 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-09-13T11:52:48.742 INFO:teuthology.misc.health.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:48.742 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:48.742 INFO:teuthology.misc.health.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:48.742 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:48.742 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:48.743 INFO:teuthology.misc.health.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:48.743 INFO:teuthology.misc.health.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:48.743 INFO:teuthology.misc.health.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:48.743 INFO:teuthology.misc.health.smithi100.stderr: 2024-09-13T11:52:48.743 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22adffb640 4 set_mon_vals no callback set 2024-09-13T11:52:48.743 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f2297fff640 1 -- 172.21.15.100:0/473744406 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f22a801b070 con 0x7f22b013cb20 2024-09-13T11:52:48.743 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22b5abe640 10 monclient: _renew_subs 2024-09-13T11:52:48.743 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22b5abe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:52:48.743 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.723+0000 7f22b5abe640 1 -- 172.21.15.100:0/473744406 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f22b01cd5e0 con 0x7f22b013cb20 2024-09-13T11:52:48.743 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.724+0000 7f2297fff640 1 -- 172.21.15.100:0/473744406 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f22a8004610 con 0x7f22b013cb20 2024-09-13T11:52:48.743 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.724+0000 7f22b5abe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:48.743 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.724+0000 7f22b5abe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:48.743 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.724+0000 7f22b5abe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:52:48.743 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.724+0000 7f22b5abe640 1 -- 172.21.15.100:0/473744406 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22b0121280 con 0x7f22b013cb20 2024-09-13T11:52:48.744 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.726+0000 7f2297fff640 1 -- 172.21.15.100:0/473744406 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f22b0121280 con 0x7f22b013cb20 2024-09-13T11:52:48.744 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.726+0000 7f2297fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:48.744 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.726+0000 7f2297fff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:52:48.787 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.786+0000 7f22b5abe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:52:48.787 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.786+0000 7f22b5abe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:52:48.787 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.786+0000 7f22b5abe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:52:48.787 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.786+0000 7f22b5abe640 1 -- 172.21.15.100:0/473744406 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f22b0137f30 con 0x7f22b013cb20 2024-09-13T11:52:48.788 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.786+0000 7f2297fff640 1 -- 172.21.15.100:0/473744406 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f22a8022070 con 0x7f22b013cb20 2024-09-13T11:52:48.788 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.786+0000 7f2297fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:52:48.788 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.786+0000 7f2297fff640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:52:48.788 INFO:teuthology.misc.health.smithi100.stdout: 2024-09-13T11:52:48.788 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":8,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","created":"2024-09-13T11:52:46.158794+0000","modified":"2024-09-13T11:52:47.800207+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T11:52:47.800207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ac784459-dbff-42ed-941e-70c741392dcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8d8d39e7-e8f1-499a-a01b-e7ba49844eae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e0bd3e30-0ff3-4067-83d3-ef8d84b29097","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"eca4429b-38cd-488a-b31d-c5856d701bde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b7ddc8f1-38d5-496a-9bf7-181eab7e5516","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9010f8e0-c97b-451c-8b29-3ccb99e982a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:52:48.789 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.787+0000 7f22b5abe640 10 monclient: shutdown 2024-09-13T11:52:48.789 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.787+0000 7f22b5abe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:52:48.789 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.787+0000 7f22b5abe640 1 -- 172.21.15.100:0/473744406 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22b013cb20 msgr2=0x7f22b0071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:48.790 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.787+0000 7f22b5abe640 1 --2- 172.21.15.100:0/473744406 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22b013cb20 0x7f22b0071300 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f22a800ef90 tx=0x7f22a800c490 comp rx=0 tx=0).stop 2024-09-13T11:52:48.790 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.787+0000 7f22b5abe640 1 -- 172.21.15.100:0/473744406 shutdown_connections 2024-09-13T11:52:48.790 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.787+0000 7f22b5abe640 1 --2- 172.21.15.100:0/473744406 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22b013cb20 0x7f22b0071300 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:48.790 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.787+0000 7f22b5abe640 1 --2- 172.21.15.100:0/473744406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22b0137b50 0x7f22b0136a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:48.790 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.787+0000 7f22b5abe640 1 --2- 172.21.15.100:0/473744406 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f22b00a58a0 0x7f22b01cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:48.790 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.787+0000 7f22b5abe640 1 -- 172.21.15.100:0/473744406 >> 172.21.15.100:0/473744406 conn(0x7f22b00a7040 msgr2=0x7f22b009fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:52:48.790 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.787+0000 7f22b5abe640 1 -- 172.21.15.100:0/473744406 shutdown_connections 2024-09-13T11:52:48.790 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:48.787+0000 7f22b5abe640 1 -- 172.21.15.100:0/473744406 wait complete. 2024-09-13T11:52:48.797 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-13T11:52:49.046 INFO:tasks.ceph.osd.2.smithi100.stderr:2024-09-13T11:52:49.044+0000 7fd138943640 -1 Falling back to public interface 2024-09-13T11:52:49.049 INFO:tasks.ceph.osd.5.smithi122.stderr:2024-09-13T11:52:49.047+0000 7fc1c721d640 -1 Falling back to public interface 2024-09-13T11:52:49.276 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T11:52:49.274+0000 7fd090534640 -1 Falling back to public interface 2024-09-13T11:52:49.536 INFO:tasks.ceph.osd.3.smithi122.stderr:2024-09-13T11:52:49.534+0000 7f5373741640 -1 Falling back to public interface 2024-09-13T11:52:49.542 INFO:tasks.ceph.osd.4.smithi122.stderr:2024-09-13T11:52:49.540+0000 7f0508dc1640 -1 Falling back to public interface 2024-09-13T11:52:49.777 INFO:tasks.ceph.osd.0.smithi100.stderr:2024-09-13T11:52:49.775+0000 7fa4ab9c3640 -1 Falling back to public interface 2024-09-13T11:52:52.213 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-13T11:52:52.210+0000 7fa1e834a640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-13T11:52:52.213 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-13T11:52:52.211+0000 7fa1e834a640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-13T11:52:52.559 INFO:tasks.ceph.osd.2.smithi100.stderr:2024-09-13T11:52:52.558+0000 7fd138943640 -1 osd.2 0 log_to_monitors true 2024-09-13T11:52:52.797 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T11:52:52.795+0000 7fd090534640 -1 osd.1 0 log_to_monitors true 2024-09-13T11:52:52.884 INFO:tasks.ceph.osd.5.smithi122.stderr:2024-09-13T11:52:52.882+0000 7fc1c721d640 -1 osd.5 0 log_to_monitors true 2024-09-13T11:52:53.328 INFO:tasks.ceph.osd.3.smithi122.stderr:2024-09-13T11:52:53.326+0000 7f5373741640 -1 osd.3 0 log_to_monitors true 2024-09-13T11:52:53.623 INFO:tasks.ceph.osd.4.smithi122.stderr:2024-09-13T11:52:53.621+0000 7f0508dc1640 -1 osd.4 0 log_to_monitors true 2024-09-13T11:52:53.772 INFO:tasks.ceph.osd.0.smithi100.stderr:2024-09-13T11:52:53.771+0000 7fa4ab9c3640 -1 osd.0 0 log_to_monitors true 2024-09-13T11:52:54.224 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T11:52:54.223+0000 7fd08c4c3640 -1 osd.1 0 waiting for initial osdmap 2024-09-13T11:52:54.230 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T11:52:54.229+0000 7fd0882ea640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:52:54.761 INFO:tasks.ceph.osd.2.smithi100.stderr:2024-09-13T11:52:54.760+0000 7fd1350e6640 -1 osd.2 0 waiting for initial osdmap 2024-09-13T11:52:54.762 INFO:tasks.ceph.osd.5.smithi122.stderr:2024-09-13T11:52:54.760+0000 7fc1c33fc640 -1 osd.5 0 waiting for initial osdmap 2024-09-13T11:52:54.772 INFO:tasks.ceph.osd.2.smithi100.stderr:2024-09-13T11:52:54.771+0000 7fd12c6f1640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:52:54.774 INFO:tasks.ceph.osd.5.smithi122.stderr:2024-09-13T11:52:54.772+0000 7fc1baa07640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:52:55.403 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T11:52:55.505 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.504+0000 7f05512cd640 10 monclient: get_monmap_and_config 2024-09-13T11:52:55.505 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.504+0000 7f05512cd640 10 monclient: build_initial_monmap 2024-09-13T11:52:55.505 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.504+0000 7f05512cd640 10 monclient: monmap: 2024-09-13T11:52:55.505 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-09-13T11:52:55.505 INFO:teuthology.misc.health.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:55.506 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-09-13T11:52:55.504933+0000 2024-09-13T11:52:55.506 INFO:teuthology.misc.health.smithi100.stderr:created 2024-09-13T11:52:55.504933+0000 2024-09-13T11:52:55.506 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:52:55.506 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:55.506 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:52:55.506 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:52:55.506 INFO:teuthology.misc.health.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:52:55.506 INFO:teuthology.misc.health.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:52:55.506 INFO:teuthology.misc.health.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:52:55.506 INFO:teuthology.misc.health.smithi100.stderr: 2024-09-13T11:52:55.514 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.504+0000 7f05512cd640 1 Processor -- start 2024-09-13T11:52:55.515 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.504+0000 7f05512cd640 1 -- start start 2024-09-13T11:52:55.515 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.504+0000 7f05512cd640 10 monclient: init 2024-09-13T11:52:55.515 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f05512cd640 10 monclient: _reopen_session rank -1 2024-09-13T11:52:55.515 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f05512cd640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-13T11:52:55.515 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f05512cd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f054c137b50 addr v1:172.21.15.122:6789/0 2024-09-13T11:52:55.515 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f05512cd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f054c0a58a0 0x7f054c0a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:55.515 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f05512cd640 10 monclient(hunting): picked mon.noname-c con 0x7f054c0a58a0 addr v2:172.21.15.122:3300/0 2024-09-13T11:52:55.515 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f05512cd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f054c13cb20 0x7f054c13ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:55.515 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f05512cd640 10 monclient(hunting): picked mon.noname-b con 0x7f054c13cb20 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:52:55.516 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f05512cd640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f054c0a5620 con 0x7f054c137b50 2024-09-13T11:52:55.516 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f05512cd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:55.516 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f05512cd640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f054c0a5ce0 con 0x7f054c0a58a0 2024-09-13T11:52:55.516 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f05512cd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:55.516 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f05512cd640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f054c0a6460 con 0x7f054c13cb20 2024-09-13T11:52:55.516 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f05512cd640 10 monclient(hunting): _renew_subs 2024-09-13T11:52:55.516 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f05512cd640 10 monclient(hunting): authenticate will time out at 756.098474s 2024-09-13T11:52:55.516 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f054b7fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f054c13cb20 0x7f054c13ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:55.516 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f054b7fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f054c13cb20 0x7f054c13ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:49062/0 (socket says 172.21.15.100:49062) 2024-09-13T11:52:55.516 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f054b7fe640 1 -- 172.21.15.100:0/2652520266 learned_addr learned my addr 172.21.15.100:0/2652520266 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:52:55.516 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f054b7fe640 10 monclient(hunting): get_auth_request con 0x7f054c13cb20 auth_method 0 2024-09-13T11:52:55.516 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f054b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:55.516 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f054b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:55.517 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f054b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:55.517 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f0543fff640 1 --2- 172.21.15.100:0/2652520266 >> v2:172.21.15.122:3300/0 conn(0x7f054c0a58a0 0x7f054c0a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:55.517 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f0543fff640 10 monclient(hunting): get_auth_request con 0x7f054c0a58a0 auth_method 0 2024-09-13T11:52:55.517 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f0543fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:55.517 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f0543fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:55.517 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f0543fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:55.517 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f054b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:55.517 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f054b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:55.517 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f054b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:55.517 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f0543fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:55.517 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f0543fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:55.517 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f0543fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:55.517 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.505+0000 7f054b7fe640 10 monclient(hunting): handle_auth_done global_id 4178 payload 274 2024-09-13T11:52:55.518 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f054b7fe640 1 -- 172.21.15.100:0/2652520266 >> v2:172.21.15.122:3300/0 conn(0x7f054c0a58a0 msgr2=0x7f054c0a0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:52:55.518 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f054b7fe640 1 --2- 172.21.15.100:0/2652520266 >> v2:172.21.15.122:3300/0 conn(0x7f054c0a58a0 0x7f054c0a0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:55.518 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f054b7fe640 1 -- 172.21.15.100:0/2652520266 >> v1:172.21.15.122:6789/0 conn(0x7f054c137b50 legacy=0x7f054c137f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:55.518 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f054b7fe640 10 monclient: _finish_hunting 0 2024-09-13T11:52:55.518 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f054b7fe640 1 monclient: found mon.noname-b 2024-09-13T11:52:55.518 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f054b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:52:55.518 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f054b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:52:55.518 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f054b7fe640 1 -- 172.21.15.100:0/2652520266 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f054c0a6940 con 0x7f054c13cb20 2024-09-13T11:52:55.518 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f0543fff640 1 --2- 172.21.15.100:0/2652520266 >> v2:172.21.15.122:3300/0 conn(0x7f054c0a58a0 0x7f054c0a0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:52:55.518 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f054b7fe640 1 --2- 172.21.15.100:0/2652520266 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f054c13cb20 0x7f054c13ef10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0544009e50 tx=0x7f054402a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c00c856eebe972ec server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:52:55.518 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f054a7fc640 1 -- 172.21.15.100:0/2652520266 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0544036070 con 0x7f054c13cb20 2024-09-13T11:52:55.518 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f054a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:55.518 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f054a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:52:55.519 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f054a7fc640 10 monclient: dump: 2024-09-13T11:52:55.519 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-09-13T11:52:55.519 INFO:teuthology.misc.health.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:55.519 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:55.519 INFO:teuthology.misc.health.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:55.519 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:55.519 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:55.519 INFO:teuthology.misc.health.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:55.519 INFO:teuthology.misc.health.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:55.519 INFO:teuthology.misc.health.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:55.519 INFO:teuthology.misc.health.smithi100.stderr: 2024-09-13T11:52:55.519 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f054a7fc640 10 monclient: _finish_auth 0 2024-09-13T11:52:55.519 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f054a7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:52:55.519 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f054a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:52:55.520 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f05512cd640 5 monclient: authenticate success, global_id 4178 2024-09-13T11:52:55.520 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f054a7fc640 1 -- 172.21.15.100:0/2652520266 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0544031600 con 0x7f054c13cb20 2024-09-13T11:52:55.520 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f05512cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:52:55.520 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f054a7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:52:55.520 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f05512cd640 10 monclient: get_monmap_and_config success 2024-09-13T11:52:55.520 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f05512cd640 4 set_mon_vals no callback set 2024-09-13T11:52:55.520 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f054a7fc640 1 -- 172.21.15.100:0/2652520266 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05440319d0 con 0x7f054c13cb20 2024-09-13T11:52:55.520 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f054a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:55.520 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f054a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:52:55.520 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.506+0000 7f054a7fc640 10 monclient: dump: 2024-09-13T11:52:55.520 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-09-13T11:52:55.520 INFO:teuthology.misc.health.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:55.520 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:55.521 INFO:teuthology.misc.health.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:55.521 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:55.521 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:55.521 INFO:teuthology.misc.health.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:55.521 INFO:teuthology.misc.health.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:55.521 INFO:teuthology.misc.health.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:55.521 INFO:teuthology.misc.health.smithi100.stderr: 2024-09-13T11:52:55.521 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.507+0000 7f05512cd640 10 monclient: shutdown 2024-09-13T11:52:55.521 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.507+0000 7f05512cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:52:55.521 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.507+0000 7f05512cd640 1 -- 172.21.15.100:0/2652520266 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f054c13cb20 msgr2=0x7f054c13ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:55.521 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.507+0000 7f05512cd640 1 --2- 172.21.15.100:0/2652520266 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f054c13cb20 0x7f054c13ef10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0544009e50 tx=0x7f054402a5f0 comp rx=0 tx=0).stop 2024-09-13T11:52:55.521 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.507+0000 7f05512cd640 1 -- 172.21.15.100:0/2652520266 shutdown_connections 2024-09-13T11:52:55.521 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.507+0000 7f05512cd640 1 --2- 172.21.15.100:0/2652520266 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f054c13cb20 0x7f054c13ef10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:55.522 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.507+0000 7f05512cd640 1 --2- 172.21.15.100:0/2652520266 >> v2:172.21.15.122:3300/0 conn(0x7f054c0a58a0 0x7f054c0a0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:55.522 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.507+0000 7f05512cd640 1 -- 172.21.15.100:0/2652520266 >> 172.21.15.100:0/2652520266 conn(0x7f054c0a7040 msgr2=0x7f054c0a7450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:52:55.522 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.507+0000 7f05512cd640 1 -- 172.21.15.100:0/2652520266 shutdown_connections 2024-09-13T11:52:55.522 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.507+0000 7f05512cd640 1 -- 172.21.15.100:0/2652520266 wait complete. 2024-09-13T11:52:55.522 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.508+0000 7f05512cd640 10 monclient: build_initial_monmap 2024-09-13T11:52:55.522 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.508+0000 7f05512cd640 10 monclient: monmap: 2024-09-13T11:52:55.522 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-09-13T11:52:55.522 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:52:55.522 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-09-13T11:52:55.522 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-09-13T11:52:55.522 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:52:55.522 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:55.522 INFO:teuthology.misc.health.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:52:55.523 INFO:teuthology.misc.health.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:52:55.523 INFO:teuthology.misc.health.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:52:55.523 INFO:teuthology.misc.health.smithi100.stderr: 2024-09-13T11:52:55.523 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.508+0000 7f05512cd640 1 Processor -- start 2024-09-13T11:52:55.523 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.508+0000 7f05512cd640 1 -- start start 2024-09-13T11:52:55.523 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.508+0000 7f05512cd640 10 monclient: init 2024-09-13T11:52:55.523 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f05512cd640 10 monclient: _reopen_session rank -1 2024-09-13T11:52:55.523 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f05512cd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:52:55.523 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f05512cd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f054c0a58a0 0x7f054c1cc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:55.523 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f05512cd640 10 monclient(hunting): picked mon.noname-c con 0x7f054c0a58a0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:52:55.523 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f05512cd640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f054c137b50 0x7f054c1369f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:55.523 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f05512cd640 10 monclient(hunting): picked mon.noname-a con 0x7f054c137b50 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:52:55.523 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f05512cd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f054c13cb20 0x7f054c071340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:55.523 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f05512cd640 10 monclient(hunting): picked mon.noname-b con 0x7f054c13cb20 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:52:55.524 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f05512cd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:55.524 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f05512cd640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f054c13f450 con 0x7f054c137b50 2024-09-13T11:52:55.524 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f05512cd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:55.524 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f05512cd640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f054c138640 con 0x7f054c13cb20 2024-09-13T11:52:55.524 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f05512cd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:52:55.524 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f05512cd640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f054c13f5d0 con 0x7f054c0a58a0 2024-09-13T11:52:55.524 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f05512cd640 10 monclient(hunting): _renew_subs 2024-09-13T11:52:55.524 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f05512cd640 10 monclient(hunting): authenticate will time out at 756.102289s 2024-09-13T11:52:55.524 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f054affd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f054c0a58a0 0x7f054c1cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:55.524 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f0543fff640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f054c137b50 0x7f054c1369f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:55.524 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f0543fff640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f054c137b50 0x7f054c1369f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:51372/0 (socket says 172.21.15.100:51372) 2024-09-13T11:52:55.524 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f054affd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f054c0a58a0 0x7f054c1cc750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:49070/0 (socket says 172.21.15.100:49070) 2024-09-13T11:52:55.524 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f0543fff640 1 -- 172.21.15.100:0/4193969593 learned_addr learned my addr 172.21.15.100:0/4193969593 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:52:55.525 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f0543fff640 10 monclient(hunting): get_auth_request con 0x7f054c137b50 auth_method 0 2024-09-13T11:52:55.525 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f0543fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:55.525 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f0543fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:55.525 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f054b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f054c13cb20 0x7f054c071340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:55.525 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f0543fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:55.525 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f054b7fe640 10 monclient(hunting): get_auth_request con 0x7f054c13cb20 auth_method 0 2024-09-13T11:52:55.525 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f054b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:55.525 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f054b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:55.525 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f054b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:55.525 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f054affd640 10 monclient(hunting): get_auth_request con 0x7f054c0a58a0 auth_method 0 2024-09-13T11:52:55.525 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f054affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:52:55.525 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f054affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:52:55.525 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f054affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:52:55.526 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f0543fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:55.526 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f0543fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:55.526 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f0543fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:55.526 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f054b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:55.526 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f054b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:55.526 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f054b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:55.526 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f054affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:52:55.526 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f054affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:52:55.526 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f054affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:52:55.526 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f0543fff640 10 monclient(hunting): handle_auth_done global_id 4200 payload 824 2024-09-13T11:52:55.526 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f0543fff640 1 -- 172.21.15.100:0/4193969593 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f054c0a58a0 msgr2=0x7f054c1cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:55.526 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f0543fff640 1 --2- 172.21.15.100:0/4193969593 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f054c0a58a0 0x7f054c1cc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:55.526 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f0543fff640 1 -- 172.21.15.100:0/4193969593 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f054c13cb20 msgr2=0x7f054c071340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:55.526 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f0543fff640 1 --2- 172.21.15.100:0/4193969593 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f054c13cb20 0x7f054c071340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:55.527 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f0543fff640 10 monclient: _finish_hunting 0 2024-09-13T11:52:55.527 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f0543fff640 1 monclient: found mon.noname-a 2024-09-13T11:52:55.527 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f0543fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:52:55.527 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f0543fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:52:55.527 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f0543fff640 1 -- 172.21.15.100:0/4193969593 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f054c069290 con 0x7f054c137b50 2024-09-13T11:52:55.527 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f054b7fe640 1 --2- 172.21.15.100:0/4193969593 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f054c13cb20 0x7f054c071340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:52:55.527 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.509+0000 7f0543fff640 1 --2- 172.21.15.100:0/4193969593 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f054c137b50 0x7f054c1369f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f053400ebe0 tx=0x7f053400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:52:55.527 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.510+0000 7f05437fe640 1 -- 172.21.15.100:0/4193969593 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0534017070 con 0x7f054c137b50 2024-09-13T11:52:55.527 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.510+0000 7f05437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:55.527 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.510+0000 7f05437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:52:55.527 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.510+0000 7f05437fe640 10 monclient: dump: 2024-09-13T11:52:55.527 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-09-13T11:52:55.527 INFO:teuthology.misc.health.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:55.527 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:55.528 INFO:teuthology.misc.health.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:55.528 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:55.528 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:55.528 INFO:teuthology.misc.health.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:55.528 INFO:teuthology.misc.health.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:55.528 INFO:teuthology.misc.health.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:55.528 INFO:teuthology.misc.health.smithi100.stderr: 2024-09-13T11:52:55.528 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.510+0000 7f05437fe640 10 monclient: _finish_auth 0 2024-09-13T11:52:55.528 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.510+0000 7f05437fe640 10 monclient: _check_auth_tickets 2024-09-13T11:52:55.528 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.510+0000 7f05437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:52:55.528 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.510+0000 7f05437fe640 1 -- 172.21.15.100:0/4193969593 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f053400cc30 con 0x7f054c137b50 2024-09-13T11:52:55.528 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.510+0000 7f05512cd640 5 monclient: authenticate success, global_id 4200 2024-09-13T11:52:55.528 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.510+0000 7f05512cd640 10 monclient: _renew_subs 2024-09-13T11:52:55.529 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.510+0000 7f05512cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:52:55.529 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.510+0000 7f05512cd640 1 -- 172.21.15.100:0/4193969593 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f054c13f020 con 0x7f054c137b50 2024-09-13T11:52:55.529 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.510+0000 7f05437fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:52:55.529 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.510+0000 7f0549ffb640 4 set_mon_vals no callback set 2024-09-13T11:52:55.529 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.510+0000 7f05512cd640 10 monclient: _renew_subs 2024-09-13T11:52:55.529 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.510+0000 7f05437fe640 1 -- 172.21.15.100:0/4193969593 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0534013470 con 0x7f054c137b50 2024-09-13T11:52:55.529 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.510+0000 7f05512cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:52:55.529 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.510+0000 7f05512cd640 1 -- 172.21.15.100:0/4193969593 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f054c120dc0 con 0x7f054c137b50 2024-09-13T11:52:55.529 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.510+0000 7f05437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:52:55.529 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.510+0000 7f05437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:52:55.529 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.510+0000 7f05437fe640 10 monclient: dump: 2024-09-13T11:52:55.529 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-09-13T11:52:55.529 INFO:teuthology.misc.health.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:52:55.529 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:55.530 INFO:teuthology.misc.health.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:52:55.530 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:52:55.530 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-09-13T11:52:55.530 INFO:teuthology.misc.health.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:52:55.530 INFO:teuthology.misc.health.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:52:55.530 INFO:teuthology.misc.health.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:52:55.530 INFO:teuthology.misc.health.smithi100.stderr: 2024-09-13T11:52:55.530 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.511+0000 7f05437fe640 1 -- 172.21.15.100:0/4193969593 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 4) ==== 50363+0+0 (secure 0 0 0) 0x7f053401a070 con 0x7f054c137b50 2024-09-13T11:52:55.530 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.511+0000 7f05437fe640 1 --2- 172.21.15.100:0/4193969593 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f052c03df10 0x7f052c0403d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:52:55.530 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.511+0000 7f05437fe640 1 -- 172.21.15.100:0/4193969593 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(11..11 src has 1..11) ==== 2767+0+0 (secure 0 0 0) 0x7f0534056ca0 con 0x7f054c137b50 2024-09-13T11:52:55.530 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.511+0000 7f054affd640 1 --2- 172.21.15.100:0/4193969593 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f052c03df10 0x7f052c0403d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:52:55.530 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.511+0000 7f054affd640 10 monclient: get_auth_request con 0x7f052c03df10 auth_method 0 2024-09-13T11:52:55.530 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.511+0000 7f05512cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:55.530 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.511+0000 7f05512cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:55.531 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.511+0000 7f05512cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:52:55.531 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.511+0000 7f05512cd640 1 -- 172.21.15.100:0/4193969593 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f054c0a8ac0 con 0x7f054c137b50 2024-09-13T11:52:55.531 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.512+0000 7f054affd640 1 --2- 172.21.15.100:0/4193969593 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f052c03df10 0x7f052c0403d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f053c000f60 tx=0x7f053c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:52:55.531 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.516+0000 7f05437fe640 1 -- 172.21.15.100:0/4193969593 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f054c0a8ac0 con 0x7f054c137b50 2024-09-13T11:52:55.531 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.516+0000 7f05437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:52:55.532 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.516+0000 7f05437fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:52:55.800 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.798+0000 7f05512cd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:52:55.800 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.798+0000 7f05512cd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:52:55.800 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.798+0000 7f05512cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:52:55.800 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.798+0000 7f05512cd640 1 -- 172.21.15.100:0/4193969593 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f054c1cd640 con 0x7f054c137b50 2024-09-13T11:52:55.800 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.798+0000 7f05437fe640 1 -- 172.21.15.100:0/4193969593 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) ==== 74+0+7075 (secure 0 0 0) 0x7f0534010cc0 con 0x7f054c137b50 2024-09-13T11:52:55.800 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.799+0000 7f05437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:52:55.800 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.799+0000 7f05437fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:52:55.801 INFO:teuthology.misc.health.smithi100.stdout: 2024-09-13T11:52:55.801 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":11,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","created":"2024-09-13T11:52:46.158794+0000","modified":"2024-09-13T11:52:55.210597+0000","last_up_change":"2024-09-13T11:52:55.210597+0000","last_in_change":"2024-09-13T11:52:47.800207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ac784459-dbff-42ed-941e-70c741392dcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8d8d39e7-e8f1-499a-a01b-e7ba49844eae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":1385319817},{"type":"v1","addr":"172.21.15.100:6809","nonce":1385319817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":1385319817},{"type":"v1","addr":"172.21.15.100:6811","nonce":1385319817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":1385319817},{"type":"v1","addr":"172.21.15.100:6815","nonce":1385319817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":1385319817},{"type":"v1","addr":"172.21.15.100:6813","nonce":1385319817}]},"public_addr":"172.21.15.100:6809/1385319817","cluster_addr":"172.21.15.100:6811/1385319817","heartbeat_back_addr":"172.21.15.100:6815/1385319817","heartbeat_front_addr":"172.21.15.100:6813/1385319817","state":["exists","up"]},{"osd":2,"uuid":"e0bd3e30-0ff3-4067-83d3-ef8d84b29097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":2354314235},{"type":"v1","addr":"172.21.15.100:6801","nonce":2354314235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":2354314235},{"type":"v1","addr":"172.21.15.100:6803","nonce":2354314235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":2354314235},{"type":"v1","addr":"172.21.15.100:6807","nonce":2354314235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":2354314235},{"type":"v1","addr":"172.21.15.100:6805","nonce":2354314235}]},"public_addr":"172.21.15.100:6801/2354314235","cluster_addr":"172.21.15.100:6803/2354314235","heartbeat_back_addr":"172.21.15.100:6807/2354314235","heartbeat_front_addr":"172.21.15.100:6805/2354314235","state":["exists","up"]},{"osd":3,"uuid":"eca4429b-38cd-488a-b31d-c5856d701bde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b7ddc8f1-38d5-496a-9bf7-181eab7e5516","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9010f8e0-c97b-451c-8b29-3ccb99e982a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":904725860},{"type":"v1","addr":"172.21.15.122:6801","nonce":904725860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":904725860},{"type":"v1","addr":"172.21.15.122:6803","nonce":904725860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":904725860},{"type":"v1","addr":"172.21.15.122:6807","nonce":904725860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":904725860},{"type":"v1","addr":"172.21.15.122:6805","nonce":904725860}]},"public_addr":"172.21.15.122:6801/904725860","cluster_addr":"172.21.15.122:6803/904725860","heartbeat_back_addr":"172.21.15.122:6807/904725860","heartbeat_front_addr":"172.21.15.122:6805/904725860","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:52:55.804 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.802+0000 7f05512cd640 1 -- 172.21.15.100:0/4193969593 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f052c03df10 msgr2=0x7f052c0403d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:55.804 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.802+0000 7f05512cd640 1 --2- 172.21.15.100:0/4193969593 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f052c03df10 0x7f052c0403d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f053c000f60 tx=0x7f053c009040 comp rx=0 tx=0).stop 2024-09-13T11:52:55.804 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.802+0000 7f05512cd640 10 monclient: shutdown 2024-09-13T11:52:55.804 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.802+0000 7f05512cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:52:55.804 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.802+0000 7f05512cd640 1 -- 172.21.15.100:0/4193969593 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f054c137b50 msgr2=0x7f054c1369f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:52:55.804 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.802+0000 7f05512cd640 1 --2- 172.21.15.100:0/4193969593 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f054c137b50 0x7f054c1369f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f053400ebe0 tx=0x7f053400c520 comp rx=0 tx=0).stop 2024-09-13T11:52:55.804 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.802+0000 7f05512cd640 1 -- 172.21.15.100:0/4193969593 shutdown_connections 2024-09-13T11:52:55.804 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.802+0000 7f05512cd640 1 --2- 172.21.15.100:0/4193969593 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f052c03df10 0x7f052c0403d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:55.804 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.802+0000 7f05512cd640 1 --2- 172.21.15.100:0/4193969593 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f054c137b50 0x7f054c1369f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:55.804 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.802+0000 7f05512cd640 1 --2- 172.21.15.100:0/4193969593 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f054c13cb20 0x7f054c071340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:55.804 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.802+0000 7f05512cd640 1 --2- 172.21.15.100:0/4193969593 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f054c0a58a0 0x7f054c1cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:52:55.805 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.802+0000 7f05512cd640 1 -- 172.21.15.100:0/4193969593 >> 172.21.15.100:0/4193969593 conn(0x7f054c0a7040 msgr2=0x7f054c09fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:52:55.805 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.802+0000 7f05512cd640 1 -- 172.21.15.100:0/4193969593 shutdown_connections 2024-09-13T11:52:55.805 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:52:55.802+0000 7f05512cd640 1 -- 172.21.15.100:0/4193969593 wait complete. 2024-09-13T11:52:55.816 DEBUG:teuthology.misc:3 of 6 OSDs are up 2024-09-13T11:52:55.825 INFO:tasks.ceph.osd.0.smithi100.stderr:2024-09-13T11:52:55.823+0000 7fa4a7952640 -1 osd.0 0 waiting for initial osdmap 2024-09-13T11:52:55.843 INFO:tasks.ceph.osd.0.smithi100.stderr:2024-09-13T11:52:55.842+0000 7fa49f771640 -1 osd.0 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:52:55.953 INFO:tasks.ceph.osd.3.smithi122.stderr:2024-09-13T11:52:55.951+0000 7f536f6d0640 -1 osd.3 0 waiting for initial osdmap 2024-09-13T11:52:55.967 INFO:tasks.ceph.osd.3.smithi122.stderr:2024-09-13T11:52:55.965+0000 7f53674ef640 -1 osd.3 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:52:55.979 INFO:tasks.ceph.osd.4.smithi122.stderr:2024-09-13T11:52:55.977+0000 7f0504d50640 -1 osd.4 0 waiting for initial osdmap 2024-09-13T11:52:55.988 INFO:tasks.ceph.osd.4.smithi122.stderr:2024-09-13T11:52:55.987+0000 7f04fcb6f640 -1 osd.4 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:53:02.422 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T11:53:02.521 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.519+0000 7f67fbb4b640 10 monclient: get_monmap_and_config 2024-09-13T11:53:02.521 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.519+0000 7f67fbb4b640 10 monclient: build_initial_monmap 2024-09-13T11:53:02.521 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.519+0000 7f67fbb4b640 10 monclient: monmap: 2024-09-13T11:53:02.521 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-09-13T11:53:02.521 INFO:teuthology.misc.health.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:02.521 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-09-13T11:53:02.520694+0000 2024-09-13T11:53:02.522 INFO:teuthology.misc.health.smithi100.stderr:created 2024-09-13T11:53:02.520694+0000 2024-09-13T11:53:02.522 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:02.522 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:02.522 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:02.522 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:02.522 INFO:teuthology.misc.health.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:02.522 INFO:teuthology.misc.health.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:02.522 INFO:teuthology.misc.health.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:02.522 INFO:teuthology.misc.health.smithi100.stderr: 2024-09-13T11:53:02.530 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.520+0000 7f67fbb4b640 1 Processor -- start 2024-09-13T11:53:02.531 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.520+0000 7f67fbb4b640 1 -- start start 2024-09-13T11:53:02.531 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.520+0000 7f67fbb4b640 10 monclient: init 2024-09-13T11:53:02.531 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67fbb4b640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:02.531 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67fbb4b640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-13T11:53:02.531 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67fbb4b640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f67f4135560 0x7f67f4135940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:02.531 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67fbb4b640 10 monclient(hunting): picked mon.noname-a con 0x7f67f4135560 addr v2:172.21.15.100:3300/0 2024-09-13T11:53:02.531 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67fbb4b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f67f4135e80 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:02.531 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67fbb4b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67f40a0d90 0x7f67f40a1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:02.531 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67fbb4b640 10 monclient(hunting): picked mon.noname-b con 0x7f67f40a0d90 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:02.531 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67fbb4b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67f409f670 con 0x7f67f4135e80 2024-09-13T11:53:02.531 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67fbb4b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:02.531 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67fbb4b640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f67f406f030 con 0x7f67f4135560 2024-09-13T11:53:02.532 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67fbb4b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:02.532 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67fbb4b640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f67f40a1cb0 con 0x7f67f40a0d90 2024-09-13T11:53:02.532 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67fbb4b640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:02.532 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67fbb4b640 10 monclient(hunting): authenticate will time out at 763.114274s 2024-09-13T11:53:02.532 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67f98e7640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f67f4135560 0x7f67f4135940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:02.532 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67fa0e8640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67f40a0d90 0x7f67f40a1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:02.532 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67f98e7640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f67f4135560 0x7f67f4135940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49320/0 (socket says 172.21.15.100:49320) 2024-09-13T11:53:02.532 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67f98e7640 1 -- 172.21.15.100:0/336140385 learned_addr learned my addr 172.21.15.100:0/336140385 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:02.532 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67f98e7640 10 monclient(hunting): get_auth_request con 0x7f67f4135560 auth_method 0 2024-09-13T11:53:02.532 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67f98e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:02.532 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67f98e7640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:02.532 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67f98e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:02.532 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67fa0e8640 10 monclient(hunting): get_auth_request con 0x7f67f40a0d90 auth_method 0 2024-09-13T11:53:02.533 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67fa0e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:02.533 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67fa0e8640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:02.533 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67fa0e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:02.533 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67f98e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:02.533 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67f98e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:02.533 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67f98e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:02.533 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67fa0e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:02.533 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67fa0e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:02.534 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67fa0e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:02.534 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67f98e7640 10 monclient(hunting): handle_auth_done global_id 4212 payload 274 2024-09-13T11:53:02.534 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67f98e7640 1 -- 172.21.15.100:0/336140385 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67f40a0d90 msgr2=0x7f67f40a1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:02.534 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67f98e7640 1 --2- 172.21.15.100:0/336140385 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67f40a0d90 0x7f67f40a1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:02.534 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67f98e7640 1 -- 172.21.15.100:0/336140385 >> v1:172.21.15.122:6789/0 conn(0x7f67f4135e80 legacy=0x7f67f40a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:02.534 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67f98e7640 10 monclient: _finish_hunting 0 2024-09-13T11:53:02.534 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67f98e7640 1 monclient: found mon.noname-a 2024-09-13T11:53:02.534 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67f98e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:02.534 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67f98e7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:02.534 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67f98e7640 1 -- 172.21.15.100:0/336140385 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f40a1f90 con 0x7f67f4135560 2024-09-13T11:53:02.534 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.521+0000 7f67fa0e8640 1 --2- 172.21.15.100:0/336140385 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67f40a0d90 0x7f67f40a1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:02.534 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.522+0000 7f67f98e7640 1 --2- 172.21.15.100:0/336140385 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f67f4135560 0x7f67f4135940 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f67f0009870 tx=0x7f67f002a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ebbc551786a6f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:02.534 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.522+0000 7f67f88e5640 1 -- 172.21.15.100:0/336140385 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f0037070 con 0x7f67f4135560 2024-09-13T11:53:02.534 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.522+0000 7f67f88e5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:02.535 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.522+0000 7f67f88e5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:02.535 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.522+0000 7f67f88e5640 10 monclient: dump: 2024-09-13T11:53:02.535 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-09-13T11:53:02.535 INFO:teuthology.misc.health.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:02.535 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:02.535 INFO:teuthology.misc.health.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:02.535 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:02.535 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:02.535 INFO:teuthology.misc.health.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:02.535 INFO:teuthology.misc.health.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:02.535 INFO:teuthology.misc.health.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:02.535 INFO:teuthology.misc.health.smithi100.stderr: 2024-09-13T11:53:02.536 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.522+0000 7f67f88e5640 10 monclient: _finish_auth 0 2024-09-13T11:53:02.536 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.522+0000 7f67f88e5640 10 monclient: _check_auth_tickets 2024-09-13T11:53:02.536 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.522+0000 7f67f88e5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:02.536 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.522+0000 7f67f88e5640 1 -- 172.21.15.100:0/336140385 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67f002adf0 con 0x7f67f4135560 2024-09-13T11:53:02.536 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.522+0000 7f67fbb4b640 5 monclient: authenticate success, global_id 4212 2024-09-13T11:53:02.536 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.522+0000 7f67fbb4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:02.536 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.522+0000 7f67f88e5640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:02.536 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.522+0000 7f67fbb4b640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:02.536 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.522+0000 7f67f88e5640 1 -- 172.21.15.100:0/336140385 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f0032660 con 0x7f67f4135560 2024-09-13T11:53:02.536 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.522+0000 7f67f88e5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:02.536 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.522+0000 7f67fbb4b640 4 set_mon_vals no callback set 2024-09-13T11:53:02.536 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.522+0000 7f67f88e5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:02.537 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.522+0000 7f67f88e5640 10 monclient: dump: 2024-09-13T11:53:02.537 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-09-13T11:53:02.537 INFO:teuthology.misc.health.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:02.537 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:02.537 INFO:teuthology.misc.health.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:02.537 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:02.537 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:02.538 INFO:teuthology.misc.health.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:02.538 INFO:teuthology.misc.health.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:02.538 INFO:teuthology.misc.health.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:02.538 INFO:teuthology.misc.health.smithi100.stderr: 2024-09-13T11:53:02.538 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.523+0000 7f67fbb4b640 10 monclient: shutdown 2024-09-13T11:53:02.538 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.523+0000 7f67fbb4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:02.538 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.523+0000 7f67fbb4b640 1 -- 172.21.15.100:0/336140385 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f67f4135560 msgr2=0x7f67f4135940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:02.538 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.523+0000 7f67fbb4b640 1 --2- 172.21.15.100:0/336140385 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f67f4135560 0x7f67f4135940 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f67f0009870 tx=0x7f67f002a0a0 comp rx=0 tx=0).stop 2024-09-13T11:53:02.538 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.523+0000 7f67fbb4b640 1 -- 172.21.15.100:0/336140385 shutdown_connections 2024-09-13T11:53:02.539 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.523+0000 7f67fbb4b640 1 --2- 172.21.15.100:0/336140385 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67f40a0d90 0x7f67f40a1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:02.539 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.523+0000 7f67fbb4b640 1 --2- 172.21.15.100:0/336140385 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f67f4135560 0x7f67f4135940 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:02.539 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.523+0000 7f67fbb4b640 1 -- 172.21.15.100:0/336140385 >> 172.21.15.100:0/336140385 conn(0x7f67f4068730 msgr2=0x7f67f4068b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:02.539 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.523+0000 7f67fbb4b640 1 -- 172.21.15.100:0/336140385 shutdown_connections 2024-09-13T11:53:02.539 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.523+0000 7f67fbb4b640 1 -- 172.21.15.100:0/336140385 wait complete. 2024-09-13T11:53:02.539 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.523+0000 7f67fbb4b640 10 monclient: build_initial_monmap 2024-09-13T11:53:02.539 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.523+0000 7f67fbb4b640 10 monclient: monmap: 2024-09-13T11:53:02.539 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-09-13T11:53:02.539 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:02.540 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:02.540 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-09-13T11:53:02.540 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:02.540 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:02.540 INFO:teuthology.misc.health.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:02.541 INFO:teuthology.misc.health.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:02.541 INFO:teuthology.misc.health.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:02.541 INFO:teuthology.misc.health.smithi100.stderr: 2024-09-13T11:53:02.541 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fbb4b640 1 Processor -- start 2024-09-13T11:53:02.541 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fbb4b640 1 -- start start 2024-09-13T11:53:02.541 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fbb4b640 10 monclient: init 2024-09-13T11:53:02.542 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fbb4b640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:02.542 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fbb4b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:53:02.542 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fbb4b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f67f40a0d90 0x7f67f41cca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:02.542 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fbb4b640 10 monclient(hunting): picked mon.noname-b con 0x7f67f40a0d90 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:02.542 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fbb4b640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f67f4135560 0x7f67f41ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:02.542 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fbb4b640 10 monclient(hunting): picked mon.noname-a con 0x7f67f4135560 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:02.542 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fbb4b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67f4135e80 0x7f67f409d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:02.542 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fbb4b640 10 monclient(hunting): picked mon.noname-c con 0x7f67f4135e80 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:02.542 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fbb4b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:02.542 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fbb4b640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f67f40694c0 con 0x7f67f4135560 2024-09-13T11:53:02.542 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fbb4b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:02.542 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fbb4b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f67f4136290 con 0x7f67f40a0d90 2024-09-13T11:53:02.543 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fbb4b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:02.543 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fbb4b640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f67f4069640 con 0x7f67f4135e80 2024-09-13T11:53:02.543 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fbb4b640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:02.543 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fbb4b640 10 monclient(hunting): authenticate will time out at 763.117678s 2024-09-13T11:53:02.543 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67f90e6640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f67f4135560 0x7f67f41ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:02.543 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fa0e8640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67f4135e80 0x7f67f409d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:02.543 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fa0e8640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67f4135e80 0x7f67f409d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47164/0 (socket says 172.21.15.100:47164) 2024-09-13T11:53:02.543 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67f90e6640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f67f4135560 0x7f67f41ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49330/0 (socket says 172.21.15.100:49330) 2024-09-13T11:53:02.543 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fa0e8640 1 -- 172.21.15.100:0/761571358 learned_addr learned my addr 172.21.15.100:0/761571358 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:02.543 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fa0e8640 10 monclient(hunting): get_auth_request con 0x7f67f4135e80 auth_method 0 2024-09-13T11:53:02.543 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fa0e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:02.543 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fa0e8640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:02.543 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fa0e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:02.544 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67f90e6640 10 monclient(hunting): get_auth_request con 0x7f67f4135560 auth_method 0 2024-09-13T11:53:02.544 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67f90e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:02.544 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67f90e6640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:02.544 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67f90e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:02.544 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fa0e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:02.544 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fa0e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:02.544 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67fa0e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:02.544 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67f90e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:02.544 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67f90e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:02.544 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.524+0000 7f67f90e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:02.544 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67f98e7640 1 --2- 172.21.15.100:0/761571358 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f67f40a0d90 0x7f67f41cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:02.544 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67f98e7640 10 monclient(hunting): get_auth_request con 0x7f67f40a0d90 auth_method 0 2024-09-13T11:53:02.544 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67f98e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:02.545 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67f98e7640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:02.545 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67f98e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:02.545 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67f90e6640 10 monclient(hunting): handle_auth_done global_id 4215 payload 824 2024-09-13T11:53:02.545 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67f90e6640 1 -- 172.21.15.100:0/761571358 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67f4135e80 msgr2=0x7f67f409d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:02.545 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67f90e6640 1 --2- 172.21.15.100:0/761571358 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67f4135e80 0x7f67f409d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:02.545 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67f90e6640 1 -- 172.21.15.100:0/761571358 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f67f40a0d90 msgr2=0x7f67f41cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:02.545 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67f90e6640 1 --2- 172.21.15.100:0/761571358 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f67f40a0d90 0x7f67f41cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:02.545 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67f90e6640 10 monclient: _finish_hunting 0 2024-09-13T11:53:02.545 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67f90e6640 1 monclient: found mon.noname-a 2024-09-13T11:53:02.545 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67f90e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:02.545 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67f90e6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:02.545 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67f90e6640 1 -- 172.21.15.100:0/761571358 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f409d070 con 0x7f67f4135560 2024-09-13T11:53:02.546 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67f90e6640 1 --2- 172.21.15.100:0/761571358 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f67f4135560 0x7f67f41ccf60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f67e400c410 tx=0x7f67e400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:02.546 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67ea7fc640 1 -- 172.21.15.100:0/761571358 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e4017070 con 0x7f67f4135560 2024-09-13T11:53:02.546 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:02.546 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67ea7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:02.546 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67ea7fc640 10 monclient: dump: 2024-09-13T11:53:02.546 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-09-13T11:53:02.546 INFO:teuthology.misc.health.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:02.546 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:02.546 INFO:teuthology.misc.health.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:02.546 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:02.546 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:02.546 INFO:teuthology.misc.health.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:02.546 INFO:teuthology.misc.health.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:02.547 INFO:teuthology.misc.health.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:02.547 INFO:teuthology.misc.health.smithi100.stderr: 2024-09-13T11:53:02.547 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67ea7fc640 10 monclient: _finish_auth 0 2024-09-13T11:53:02.547 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67ea7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:53:02.547 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:02.547 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67ea7fc640 1 -- 172.21.15.100:0/761571358 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67e4005230 con 0x7f67f4135560 2024-09-13T11:53:02.547 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67fbb4b640 5 monclient: authenticate success, global_id 4215 2024-09-13T11:53:02.547 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67fbb4b640 10 monclient: _renew_subs 2024-09-13T11:53:02.547 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67fbb4b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:02.547 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67fbb4b640 1 -- 172.21.15.100:0/761571358 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67f409ed30 con 0x7f67f4135560 2024-09-13T11:53:02.548 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67ea7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:02.548 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67ea7fc640 1 -- 172.21.15.100:0/761571358 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e40045e0 con 0x7f67f4135560 2024-09-13T11:53:02.548 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:02.548 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67ea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:02.548 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67ea7fc640 10 monclient: dump: 2024-09-13T11:53:02.548 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-09-13T11:53:02.548 INFO:teuthology.misc.health.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:02.548 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:02.548 INFO:teuthology.misc.health.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:02.548 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:02.548 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:02.548 INFO:teuthology.misc.health.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:02.548 INFO:teuthology.misc.health.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:02.549 INFO:teuthology.misc.health.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:02.549 INFO:teuthology.misc.health.smithi100.stderr: 2024-09-13T11:53:02.549 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67fbb4b640 10 monclient: _renew_subs 2024-09-13T11:53:02.549 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67fbb4b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:02.549 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67ebfff640 4 set_mon_vals no callback set 2024-09-13T11:53:02.549 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.525+0000 7f67fbb4b640 1 -- 172.21.15.100:0/761571358 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67f4127750 con 0x7f67f4135560 2024-09-13T11:53:02.549 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.526+0000 7f67ea7fc640 1 -- 172.21.15.100:0/761571358 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f67e401a070 con 0x7f67f4135560 2024-09-13T11:53:02.549 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.526+0000 7f67fbb4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:02.549 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.526+0000 7f67fbb4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:02.549 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.526+0000 7f67fbb4b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:02.549 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.526+0000 7f67fbb4b640 1 -- 172.21.15.100:0/761571358 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67f409ff80 con 0x7f67f4135560 2024-09-13T11:53:02.550 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.526+0000 7f67ea7fc640 1 --2- 172.21.15.100:0/761571358 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f67d803dfa0 0x7f67d8040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:02.550 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.526+0000 7f67ea7fc640 1 -- 172.21.15.100:0/761571358 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 3993+0+0 (secure 0 0 0) 0x7f67e4051820 con 0x7f67f4135560 2024-09-13T11:53:02.550 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.527+0000 7f67f98e7640 1 --2- 172.21.15.100:0/761571358 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f67d803dfa0 0x7f67d8040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:02.550 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.527+0000 7f67f98e7640 10 monclient: get_auth_request con 0x7f67d803dfa0 auth_method 0 2024-09-13T11:53:02.550 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.527+0000 7f67f98e7640 1 --2- 172.21.15.100:0/761571358 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f67d803dfa0 0x7f67d8040460 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f67f0034000 tx=0x7f67f002e000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:02.551 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.531+0000 7f67ea7fc640 1 -- 172.21.15.100:0/761571358 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f67e4053d10 con 0x7f67f4135560 2024-09-13T11:53:02.551 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.531+0000 7f67ea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:02.551 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.531+0000 7f67ea7fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:02.706 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.705+0000 7f67fbb4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:53:02.707 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.705+0000 7f67fbb4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:53:02.707 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.705+0000 7f67fbb4b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:02.707 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.705+0000 7f67fbb4b640 1 -- 172.21.15.100:0/761571358 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f67f41cd4a0 con 0x7f67f4135560 2024-09-13T11:53:02.707 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.706+0000 7f67ea7fc640 1 -- 172.21.15.100:0/761571358 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+10497 (secure 0 0 0) 0x7f67e40115e0 con 0x7f67f4135560 2024-09-13T11:53:02.708 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.706+0000 7f67ea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:53:02.708 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.706+0000 7f67ea7fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:02.708 INFO:teuthology.misc.health.smithi100.stdout: 2024-09-13T11:53:02.708 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":14,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","created":"2024-09-13T11:52:46.158794+0000","modified":"2024-09-13T11:52:58.239554+0000","last_up_change":"2024-09-13T11:52:56.225375+0000","last_in_change":"2024-09-13T11:52:47.800207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T11:52:55.233767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac784459-dbff-42ed-941e-70c741392dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":92840425},{"type":"v1","addr":"172.21.15.100:6817","nonce":92840425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":92840425},{"type":"v1","addr":"172.21.15.100:6819","nonce":92840425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6822","nonce":92840425},{"type":"v1","addr":"172.21.15.100:6823","nonce":92840425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":92840425},{"type":"v1","addr":"172.21.15.100:6821","nonce":92840425}]},"public_addr":"172.21.15.100:6817/92840425","cluster_addr":"172.21.15.100:6819/92840425","heartbeat_back_addr":"172.21.15.100:6823/92840425","heartbeat_front_addr":"172.21.15.100:6821/92840425","state":["exists","up"]},{"osd":1,"uuid":"8d8d39e7-e8f1-499a-a01b-e7ba49844eae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":1385319817},{"type":"v1","addr":"172.21.15.100:6809","nonce":1385319817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":1385319817},{"type":"v1","addr":"172.21.15.100:6811","nonce":1385319817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":1385319817},{"type":"v1","addr":"172.21.15.100:6815","nonce":1385319817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":1385319817},{"type":"v1","addr":"172.21.15.100:6813","nonce":1385319817}]},"public_addr":"172.21.15.100:6809/1385319817","cluster_addr":"172.21.15.100:6811/1385319817","heartbeat_back_addr":"172.21.15.100:6815/1385319817","heartbeat_front_addr":"172.21.15.100:6813/1385319817","state":["exists","up"]},{"osd":2,"uuid":"e0bd3e30-0ff3-4067-83d3-ef8d84b29097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":2354314235},{"type":"v1","addr":"172.21.15.100:6801","nonce":2354314235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":2354314235},{"type":"v1","addr":"172.21.15.100:6803","nonce":2354314235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":2354314235},{"type":"v1","addr":"172.21.15.100:6807","nonce":2354314235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":2354314235},{"type":"v1","addr":"172.21.15.100:6805","nonce":2354314235}]},"public_addr":"172.21.15.100:6801/2354314235","cluster_addr":"172.21.15.100:6803/2354314235","heartbeat_back_addr":"172.21.15.100:6807/2354314235","heartbeat_front_addr":"172.21.15.100:6805/2354314235","state":["exists","up"]},{"osd":3,"uuid":"eca4429b-38cd-488a-b31d-c5856d701bde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3206511700},{"type":"v1","addr":"172.21.15.122:6809","nonce":3206511700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3206511700},{"type":"v1","addr":"172.21.15.122:6811","nonce":3206511700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3206511700},{"type":"v1","addr":"172.21.15.122:6815","nonce":3206511700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3206511700},{"type":"v1","addr":"172.21.15.122:6813","nonce":3206511700}]},"public_addr":"172.21.15.122:6809/3206511700","cluster_addr":"172.21.15.122:6811/3206511700","heartbeat_back_addr":"172.21.15.122:6815/3206511700","heartbeat_front_addr":"172.21.15.122:6813/3206511700","state":["exists","up"]},{"osd":4,"uuid":"b7ddc8f1-38d5-496a-9bf7-181eab7e5516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":777240702},{"type":"v1","addr":"172.21.15.122:6817","nonce":777240702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":777240702},{"type":"v1","addr":"172.21.15.122:6819","nonce":777240702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":777240702},{"type":"v1","addr":"172.21.15.122:6823","nonce":777240702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":777240702},{"type":"v1","addr":"172.21.15.122:6821","nonce":777240702}]},"public_addr":"172.21.15.122:6817/777240702","cluster_addr":"172.21.15.122:6819/777240702","heartbeat_back_addr":"172.21.15.122:6823/777240702","heartbeat_front_addr":"172.21.15.122:6821/777240702","state":["exists","up"]},{"osd":5,"uuid":"9010f8e0-c97b-451c-8b29-3ccb99e982a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":904725860},{"type":"v1","addr":"172.21.15.122:6801","nonce":904725860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":904725860},{"type":"v1","addr":"172.21.15.122:6803","nonce":904725860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":904725860},{"type":"v1","addr":"172.21.15.122:6807","nonce":904725860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":904725860},{"type":"v1","addr":"172.21.15.122:6805","nonce":904725860}]},"public_addr":"172.21.15.122:6801/904725860","cluster_addr":"172.21.15.122:6803/904725860","heartbeat_back_addr":"172.21.15.122:6807/904725860","heartbeat_front_addr":"172.21.15.122:6805/904725860","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:54.752725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:53.791663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:53.535397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:54.348980+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:54.658725+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:53.932256+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:53:02.709 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.708+0000 7f67fbb4b640 1 -- 172.21.15.100:0/761571358 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f67d803dfa0 msgr2=0x7f67d8040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:02.709 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.708+0000 7f67fbb4b640 1 --2- 172.21.15.100:0/761571358 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f67d803dfa0 0x7f67d8040460 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f67f0034000 tx=0x7f67f002e000 comp rx=0 tx=0).stop 2024-09-13T11:53:02.710 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.708+0000 7f67fbb4b640 10 monclient: shutdown 2024-09-13T11:53:02.710 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.708+0000 7f67fbb4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:02.710 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.708+0000 7f67fbb4b640 1 -- 172.21.15.100:0/761571358 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f67f4135560 msgr2=0x7f67f41ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:02.710 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.708+0000 7f67fbb4b640 1 --2- 172.21.15.100:0/761571358 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f67f4135560 0x7f67f41ccf60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f67e400c410 tx=0x7f67e400c8d0 comp rx=0 tx=0).stop 2024-09-13T11:53:02.710 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.708+0000 7f67fbb4b640 1 -- 172.21.15.100:0/761571358 shutdown_connections 2024-09-13T11:53:02.710 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.708+0000 7f67fbb4b640 1 --2- 172.21.15.100:0/761571358 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f67d803dfa0 0x7f67d8040460 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:02.710 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.708+0000 7f67fbb4b640 1 --2- 172.21.15.100:0/761571358 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f67f4135560 0x7f67f41ccf60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:02.710 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.708+0000 7f67fbb4b640 1 --2- 172.21.15.100:0/761571358 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67f4135e80 0x7f67f409d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:02.710 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.708+0000 7f67fbb4b640 1 --2- 172.21.15.100:0/761571358 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f67f40a0d90 0x7f67f41cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:02.710 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.708+0000 7f67fbb4b640 1 -- 172.21.15.100:0/761571358 >> 172.21.15.100:0/761571358 conn(0x7f67f4068730 msgr2=0x7f67f411c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:02.710 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.708+0000 7f67fbb4b640 1 -- 172.21.15.100:0/761571358 shutdown_connections 2024-09-13T11:53:02.710 INFO:teuthology.misc.health.smithi100.stderr:2024-09-13T11:53:02.708+0000 7f67fbb4b640 1 -- 172.21.15.100:0/761571358 wait complete. 2024-09-13T11:53:02.719 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-13T11:53:02.719 INFO:tasks.ceph:Creating RBD pool 2024-09-13T11:53:02.719 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-13T11:53:02.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.815+0000 7f9e343d1640 10 monclient: get_monmap_and_config 2024-09-13T11:53:02.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.815+0000 7f9e343d1640 10 monclient: build_initial_monmap 2024-09-13T11:53:02.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.815+0000 7f9e343d1640 10 monclient: monmap: 2024-09-13T11:53:02.817 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:02.818 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:02.818 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:02.816434+0000 2024-09-13T11:53:02.818 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:02.816434+0000 2024-09-13T11:53:02.818 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:02.818 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:02.818 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:02.818 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:02.818 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:02.818 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:02.818 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:02.819 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:02.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.816+0000 7f9e343d1640 1 Processor -- start 2024-09-13T11:53:02.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.816+0000 7f9e343d1640 1 -- start start 2024-09-13T11:53:02.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.816+0000 7f9e343d1640 10 monclient: init 2024-09-13T11:53:02.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.816+0000 7f9e343d1640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:02.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.816+0000 7f9e343d1640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-13T11:53:02.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.816+0000 7f9e343d1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9e2c0a1650 0x7f9e2c09fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:02.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.816+0000 7f9e343d1640 10 monclient(hunting): picked mon.noname-c con 0x7f9e2c0a1650 addr v2:172.21.15.122:3300/0 2024-09-13T11:53:02.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.816+0000 7f9e343d1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e2c0a1a30 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:02.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.816+0000 7f9e343d1640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9e2c0a1e40 0x7f9e2c0a3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:02.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.816+0000 7f9e343d1640 10 monclient(hunting): picked mon.noname-a con 0x7f9e2c0a1e40 addr v2:172.21.15.100:3300/0 2024-09-13T11:53:02.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.816+0000 7f9e343d1640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e2c0742f0 con 0x7f9e2c0a1a30 2024-09-13T11:53:02.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.816+0000 7f9e343d1640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:02.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.816+0000 7f9e343d1640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f9e2c0a0250 con 0x7f9e2c0a1e40 2024-09-13T11:53:02.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.816+0000 7f9e343d1640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:02.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.816+0000 7f9e343d1640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f9e2c06f0f0 con 0x7f9e2c0a1650 2024-09-13T11:53:02.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.816+0000 7f9e343d1640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:02.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.816+0000 7f9e343d1640 10 monclient(hunting): authenticate will time out at 763.409854s 2024-09-13T11:53:02.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.816+0000 7f9e3296e640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9e2c0a1e40 0x7f9e2c0a3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:02.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.816+0000 7f9e3196c640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f9e2c0a1a30 0x7f9e2c0a36c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:57856/0 (socket says 172.21.15.100:57856) 2024-09-13T11:53:02.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.816+0000 7f9e3196c640 1 -- 172.21.15.100:0/2159122429 learned_addr learned my addr 172.21.15.100:0/2159122429 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:53:02.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.817+0000 7f9e3296e640 10 monclient(hunting): get_auth_request con 0x7f9e2c0a1e40 auth_method 0 2024-09-13T11:53:02.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.817+0000 7f9e3296e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:02.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.817+0000 7f9e3296e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:02.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.817+0000 7f9e3296e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:02.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.817+0000 7f9e3296e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:02.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.817+0000 7f9e3296e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:02.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.817+0000 7f9e3296e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:02.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.817+0000 7f9e3216d640 1 --2- 172.21.15.100:0/2159122429 >> v2:172.21.15.122:3300/0 conn(0x7f9e2c0a1650 0x7f9e2c09fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:02.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.817+0000 7f9e3216d640 10 monclient(hunting): get_auth_request con 0x7f9e2c0a1650 auth_method 0 2024-09-13T11:53:02.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.817+0000 7f9e3216d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:02.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.817+0000 7f9e3216d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:02.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.817+0000 7f9e3216d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:02.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.817+0000 7f9e3296e640 10 monclient(hunting): handle_auth_done global_id 4218 payload 274 2024-09-13T11:53:02.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.817+0000 7f9e3296e640 1 -- 172.21.15.100:0/2159122429 >> v2:172.21.15.122:3300/0 conn(0x7f9e2c0a1650 msgr2=0x7f9e2c09fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:02.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.817+0000 7f9e3296e640 1 --2- 172.21.15.100:0/2159122429 >> v2:172.21.15.122:3300/0 conn(0x7f9e2c0a1650 0x7f9e2c09fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:02.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.817+0000 7f9e3296e640 1 -- 172.21.15.100:0/2159122429 >> v1:172.21.15.100:6789/0 conn(0x7f9e2c0a1a30 legacy=0x7f9e2c0a36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:02.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.817+0000 7f9e3296e640 10 monclient: _finish_hunting 0 2024-09-13T11:53:02.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.817+0000 7f9e3296e640 1 monclient: found mon.noname-a 2024-09-13T11:53:02.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.817+0000 7f9e3296e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:02.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.817+0000 7f9e3296e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:02.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.817+0000 7f9e3296e640 1 -- 172.21.15.100:0/2159122429 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e2c070410 con 0x7f9e2c0a1e40 2024-09-13T11:53:02.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.817+0000 7f9e3216d640 1 --2- 172.21.15.100:0/2159122429 >> v2:172.21.15.122:3300/0 conn(0x7f9e2c0a1650 0x7f9e2c09fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:53:02.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.817+0000 7f9e3296e640 1 --2- 172.21.15.100:0/2159122429 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e2c0a1e40 0x7f9e2c0a3dd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c06e1e0 tx=0x7f9e2402a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ba02c5032fbc7d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:02.835 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e3116b640 1 -- 172.21.15.100:0/2159122429 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e24007740 con 0x7f9e2c0a1e40 2024-09-13T11:53:02.835 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e3116b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:02.835 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e3116b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:02.835 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e3116b640 10 monclient: dump: 2024-09-13T11:53:02.835 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:02.835 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:02.835 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:02.835 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:02.835 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:02.835 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:02.836 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:02.836 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:02.836 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:02.836 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:02.836 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e3116b640 10 monclient: _finish_auth 0 2024-09-13T11:53:02.836 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e3116b640 10 monclient: _check_auth_tickets 2024-09-13T11:53:02.836 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e3116b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:02.836 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e3116b640 1 -- 172.21.15.100:0/2159122429 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e240078e0 con 0x7f9e2c0a1e40 2024-09-13T11:53:02.836 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e343d1640 5 monclient: authenticate success, global_id 4218 2024-09-13T11:53:02.836 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e343d1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:02.836 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e3116b640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:02.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e343d1640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:02.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e343d1640 4 set_mon_vals no callback set 2024-09-13T11:53:02.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e3116b640 1 -- 172.21.15.100:0/2159122429 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e24007cb0 con 0x7f9e2c0a1e40 2024-09-13T11:53:02.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e3116b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:02.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e3116b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:02.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e3116b640 10 monclient: dump: 2024-09-13T11:53:02.837 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:02.837 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:02.837 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:02.837 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:02.837 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:02.838 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:02.838 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:02.838 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:02.838 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:02.838 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:02.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e343d1640 10 monclient: shutdown 2024-09-13T11:53:02.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e343d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:02.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e343d1640 1 -- 172.21.15.100:0/2159122429 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e2c0a1e40 msgr2=0x7f9e2c0a3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:02.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e343d1640 1 --2- 172.21.15.100:0/2159122429 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e2c0a1e40 0x7f9e2c0a3dd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c06e1e0 tx=0x7f9e2402a560 comp rx=0 tx=0).stop 2024-09-13T11:53:02.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e343d1640 1 -- 172.21.15.100:0/2159122429 shutdown_connections 2024-09-13T11:53:02.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e343d1640 1 --2- 172.21.15.100:0/2159122429 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e2c0a1e40 0x7f9e2c0a3dd0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:02.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e343d1640 1 --2- 172.21.15.100:0/2159122429 >> v2:172.21.15.122:3300/0 conn(0x7f9e2c0a1650 0x7f9e2c09fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:02.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e343d1640 1 -- 172.21.15.100:0/2159122429 >> 172.21.15.100:0/2159122429 conn(0x7f9e2c0a82e0 msgr2=0x7f9e2c0a47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:02.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e343d1640 1 -- 172.21.15.100:0/2159122429 shutdown_connections 2024-09-13T11:53:02.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.818+0000 7f9e343d1640 1 -- 172.21.15.100:0/2159122429 wait complete. 2024-09-13T11:53:02.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.819+0000 7f9e343d1640 10 monclient: build_initial_monmap 2024-09-13T11:53:02.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.819+0000 7f9e343d1640 10 monclient: monmap: 2024-09-13T11:53:02.839 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:02.840 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:02.840 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:02.840 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:02.840 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:02.840 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:02.840 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:02.841 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:02.841 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:02.841 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:02.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.819+0000 7f9e343d1640 1 Processor -- start 2024-09-13T11:53:02.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.819+0000 7f9e343d1640 1 -- start start 2024-09-13T11:53:02.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.819+0000 7f9e343d1640 10 monclient: init 2024-09-13T11:53:02.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.819+0000 7f9e343d1640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:02.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.819+0000 7f9e343d1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:53:02.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.819+0000 7f9e343d1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e2c0a1650 0x7f9e2c1ac590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:02.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e343d1640 10 monclient(hunting): picked mon.noname-b con 0x7f9e2c0a1650 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:02.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e343d1640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e2c0a1a30 0x7f9e2c1cc930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:02.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e343d1640 10 monclient(hunting): picked mon.noname-a con 0x7f9e2c0a1a30 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:02.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e343d1640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e2c0a1e40 0x7f9e2c1c6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:02.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e343d1640 10 monclient(hunting): picked mon.noname-c con 0x7f9e2c0a1e40 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:02.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e343d1640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:02.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e343d1640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2c0a52b0 con 0x7f9e2c0a1a30 2024-09-13T11:53:02.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e343d1640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:02.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e343d1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2c0a5130 con 0x7f9e2c0a1650 2024-09-13T11:53:02.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e343d1640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:02.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e343d1640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f9e2c0a4310 con 0x7f9e2c0a1e40 2024-09-13T11:53:02.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e343d1640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:02.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e343d1640 10 monclient(hunting): authenticate will time out at 763.413042s 2024-09-13T11:53:02.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3196c640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e2c0a1a30 0x7f9e2c1cc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:02.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3196c640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e2c0a1a30 0x7f9e2c1cc930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49360/0 (socket says 172.21.15.100:49360) 2024-09-13T11:53:02.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3296e640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e2c0a1e40 0x7f9e2c1c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:02.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3196c640 1 -- 172.21.15.100:0/2864379184 learned_addr learned my addr 172.21.15.100:0/2864379184 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:02.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3196c640 10 monclient(hunting): get_auth_request con 0x7f9e2c0a1a30 auth_method 0 2024-09-13T11:53:02.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3196c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:02.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3196c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:02.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3196c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:02.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3296e640 10 monclient(hunting): get_auth_request con 0x7f9e2c0a1e40 auth_method 0 2024-09-13T11:53:02.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3296e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:02.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3296e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:02.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3296e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:02.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3196c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:02.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3196c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:02.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3196c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:02.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3296e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:02.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3296e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:02.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3296e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:02.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3216d640 1 --2- 172.21.15.100:0/2864379184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e2c0a1650 0x7f9e2c1ac590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:02.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3216d640 10 monclient(hunting): get_auth_request con 0x7f9e2c0a1650 auth_method 0 2024-09-13T11:53:02.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3216d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:02.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3216d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:02.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3216d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:02.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3196c640 10 monclient(hunting): handle_auth_done global_id 4224 payload 824 2024-09-13T11:53:02.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3196c640 1 -- 172.21.15.100:0/2864379184 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e2c0a1e40 msgr2=0x7f9e2c1c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:02.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3196c640 1 --2- 172.21.15.100:0/2864379184 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e2c0a1e40 0x7f9e2c1c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:02.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3196c640 1 -- 172.21.15.100:0/2864379184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e2c0a1650 msgr2=0x7f9e2c1ac590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:02.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3196c640 1 --2- 172.21.15.100:0/2864379184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e2c0a1650 0x7f9e2c1ac590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:02.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3196c640 10 monclient: _finish_hunting 0 2024-09-13T11:53:02.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3196c640 1 monclient: found mon.noname-a 2024-09-13T11:53:02.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3196c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:02.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3196c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:02.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3196c640 1 -- 172.21.15.100:0/2864379184 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e2c1ab420 con 0x7f9e2c0a1a30 2024-09-13T11:53:02.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3196c640 1 --2- 172.21.15.100:0/2864379184 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e2c0a1a30 0x7f9e2c1cc930 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c00c420 tx=0x7f9e1c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:02.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e22ffd640 1 -- 172.21.15.100:0/2864379184 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c017070 con 0x7f9e2c0a1a30 2024-09-13T11:53:02.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:02.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e22ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:02.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e22ffd640 10 monclient: dump: 2024-09-13T11:53:02.847 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:02.847 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:02.847 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:02.847 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:02.847 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:02.847 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:02.847 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:02.847 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:02.848 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:02.848 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:02.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e22ffd640 10 monclient: _finish_auth 0 2024-09-13T11:53:02.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e22ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:53:02.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:02.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e22ffd640 1 -- 172.21.15.100:0/2864379184 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e1c005ce0 con 0x7f9e2c0a1a30 2024-09-13T11:53:02.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e343d1640 5 monclient: authenticate success, global_id 4224 2024-09-13T11:53:02.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e343d1640 10 monclient: _renew_subs 2024-09-13T11:53:02.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e343d1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:02.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e343d1640 1 -- 172.21.15.100:0/2864379184 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e2c076b20 con 0x7f9e2c0a1a30 2024-09-13T11:53:02.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e22ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:02.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e22ffd640 1 -- 172.21.15.100:0/2864379184 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c007310 con 0x7f9e2c0a1a30 2024-09-13T11:53:02.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:02.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e22ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:02.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e22ffd640 10 monclient: dump: 2024-09-13T11:53:02.849 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:02.849 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:02.849 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:02.849 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:02.849 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:02.849 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:02.849 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:02.849 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:02.850 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:02.850 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:02.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e3096a640 4 set_mon_vals no callback set 2024-09-13T11:53:02.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e343d1640 10 monclient: _renew_subs 2024-09-13T11:53:02.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e343d1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:02.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.820+0000 7f9e343d1640 1 -- 172.21.15.100:0/2864379184 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e2c1acf70 con 0x7f9e2c0a1a30 2024-09-13T11:53:02.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.821+0000 7f9e22ffd640 1 -- 172.21.15.100:0/2864379184 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f9e1c01a070 con 0x7f9e2c0a1a30 2024-09-13T11:53:02.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.821+0000 7f9e343d1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:02.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.821+0000 7f9e343d1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:02.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.821+0000 7f9e343d1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:02.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.821+0000 7f9e343d1640 1 -- 172.21.15.100:0/2864379184 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e2c09fd10 con 0x7f9e2c0a1a30 2024-09-13T11:53:02.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.821+0000 7f9e22ffd640 1 --2- 172.21.15.100:0/2864379184 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9e1003dfd0 0x7f9e10040490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:02.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.821+0000 7f9e22ffd640 1 -- 172.21.15.100:0/2864379184 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 3993+0+0 (secure 0 0 0) 0x7f9e1c051710 con 0x7f9e2c0a1a30 2024-09-13T11:53:02.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.822+0000 7f9e3216d640 1 --2- 172.21.15.100:0/2864379184 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9e1003dfd0 0x7f9e10040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:02.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.822+0000 7f9e3216d640 10 monclient: get_auth_request con 0x7f9e1003dfd0 auth_method 0 2024-09-13T11:53:02.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.822+0000 7f9e3216d640 1 --2- 172.21.15.100:0/2864379184 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9e1003dfd0 0x7f9e10040490 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9e28006fd0 tx=0x7f9e28008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:02.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.826+0000 7f9e22ffd640 1 -- 172.21.15.100:0/2864379184 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f9e1c01ada0 con 0x7f9e2c0a1a30 2024-09-13T11:53:02.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.826+0000 7f9e22ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:02.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.826+0000 7f9e22ffd640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:03.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.999+0000 7f9e343d1640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-13T11:53:03.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.999+0000 7f9e343d1640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-13T11:53:03.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.999+0000 7f9e343d1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:03.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:02.999+0000 7f9e343d1640 1 -- 172.21.15.100:0/2864379184 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7f9e2c06ca60 con 0x7f9e2c0a1a30 2024-09-13T11:53:03.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.251+0000 7f9e22ffd640 1 -- 172.21.15.100:0/2864379184 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v15) ==== 109+0+0 (secure 0 0 0) 0x7f9e1c01b8d0 con 0x7f9e2c0a1a30 2024-09-13T11:53:03.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.251+0000 7f9e22ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-13T11:53:03.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.251+0000 7f9e22ffd640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-13T11:53:03.253 INFO:teuthology.orchestra.run.smithi100.stderr:pool 'rbd' created 2024-09-13T11:53:03.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.254+0000 7f9e343d1640 1 -- 172.21.15.100:0/2864379184 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9e1003dfd0 msgr2=0x7f9e10040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:03.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.254+0000 7f9e343d1640 1 --2- 172.21.15.100:0/2864379184 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9e1003dfd0 0x7f9e10040490 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9e28006fd0 tx=0x7f9e28008040 comp rx=0 tx=0).stop 2024-09-13T11:53:03.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.254+0000 7f9e343d1640 10 monclient: shutdown 2024-09-13T11:53:03.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.254+0000 7f9e343d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:03.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.254+0000 7f9e343d1640 1 -- 172.21.15.100:0/2864379184 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e2c0a1a30 msgr2=0x7f9e2c1cc930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:03.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.254+0000 7f9e343d1640 1 --2- 172.21.15.100:0/2864379184 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e2c0a1a30 0x7f9e2c1cc930 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c00c420 tx=0x7f9e1c00c8e0 comp rx=0 tx=0).stop 2024-09-13T11:53:03.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.255+0000 7f9e343d1640 1 -- 172.21.15.100:0/2864379184 shutdown_connections 2024-09-13T11:53:03.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.255+0000 7f9e343d1640 1 --2- 172.21.15.100:0/2864379184 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9e1003dfd0 0x7f9e10040490 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:03.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.255+0000 7f9e343d1640 1 --2- 172.21.15.100:0/2864379184 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e2c0a1a30 0x7f9e2c1cc930 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:03.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.255+0000 7f9e343d1640 1 --2- 172.21.15.100:0/2864379184 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e2c0a1e40 0x7f9e2c1c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:03.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.255+0000 7f9e343d1640 1 --2- 172.21.15.100:0/2864379184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e2c0a1650 0x7f9e2c1ac590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:03.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.255+0000 7f9e343d1640 1 -- 172.21.15.100:0/2864379184 >> 172.21.15.100:0/2864379184 conn(0x7f9e2c0a82e0 msgr2=0x7f9e2c070ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:03.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.255+0000 7f9e343d1640 1 -- 172.21.15.100:0/2864379184 shutdown_connections 2024-09-13T11:53:03.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.255+0000 7f9e343d1640 1 -- 172.21.15.100:0/2864379184 wait complete. 2024-09-13T11:53:03.267 DEBUG:teuthology.orchestra.run.smithi100:> rbd --cluster ceph pool init rbd 2024-09-13T11:53:03.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.318+0000 7f036371cf00 10 monclient: get_monmap_and_config 2024-09-13T11:53:03.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.318+0000 7f036371cf00 10 monclient: build_initial_monmap 2024-09-13T11:53:03.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.319+0000 7f036371cf00 10 monclient: monmap: 2024-09-13T11:53:03.320 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:03.321 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:03.321 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:03.319895+0000 2024-09-13T11:53:03.321 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:03.319895+0000 2024-09-13T11:53:03.321 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:03.321 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:03.321 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:03.321 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:03.321 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:03.321 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:03.321 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:03.321 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:03.330 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.319+0000 7f036371cf00 1 Processor -- start 2024-09-13T11:53:03.330 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.319+0000 7f036371cf00 1 -- start start 2024-09-13T11:53:03.330 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.319+0000 7f036371cf00 10 monclient: init 2024-09-13T11:53:03.330 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036371cf00 10 monclient: _reopen_session rank -1 2024-09-13T11:53:03.330 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036371cf00 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-13T11:53:03.330 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036371cf00 10 monclient(hunting): picked mon.noname-c-legacy con 0x55757da539d0 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:03.330 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036371cf00 1 --2- >> v2:172.21.15.100:3300/0 conn(0x55757da55060 0x55757daef0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:03.330 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036371cf00 10 monclient(hunting): picked mon.noname-a con 0x55757da55060 addr v2:172.21.15.100:3300/0 2024-09-13T11:53:03.331 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036371cf00 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55757da544c0 0x55757daf1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:03.331 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036371cf00 10 monclient(hunting): picked mon.noname-b con 0x55757da544c0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:03.331 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036371cf00 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55757d8b4fe0 con 0x55757da539d0 2024-09-13T11:53:03.331 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036371cf00 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:03.331 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036371cf00 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x55757d923e50 con 0x55757da55060 2024-09-13T11:53:03.331 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036371cf00 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:03.331 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036371cf00 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x55757d9348c0 con 0x55757da544c0 2024-09-13T11:53:03.331 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036371cf00 10 monclient(hunting): _renew_subs 2024-09-13T11:53:03.331 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036371cf00 10 monclient(hunting): authenticate will time out at 763.913266s 2024-09-13T11:53:03.331 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036346a640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55757da544c0 0x55757daf1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:03.332 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036346a640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55757da544c0 0x55757daf1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47188/0 (socket says 172.21.15.100:47188) 2024-09-13T11:53:03.332 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f0361a07640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x55757da55060 0x55757daef0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:03.332 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036346a640 1 -- 172.21.15.100:0/112386678 learned_addr learned my addr 172.21.15.100:0/112386678 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:03.332 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036346a640 10 monclient(hunting): get_auth_request con 0x55757da544c0 auth_method 0 2024-09-13T11:53:03.332 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036346a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:03.332 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036346a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:03.332 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036346a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:03.332 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f0361a07640 10 monclient(hunting): get_auth_request con 0x55757da55060 auth_method 0 2024-09-13T11:53:03.332 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f0361a07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:03.333 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f0361a07640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:03.333 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f0361a07640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:03.333 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036346a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:03.333 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036346a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:03.333 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036346a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:03.333 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f0361a07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:03.333 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f0361a07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:03.333 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f0361a07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:03.333 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036346a640 10 monclient(hunting): handle_auth_done global_id 4202 payload 274 2024-09-13T11:53:03.333 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036346a640 1 -- 172.21.15.100:0/112386678 >> v2:172.21.15.100:3300/0 conn(0x55757da55060 msgr2=0x55757daef0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:03.333 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036346a640 1 --2- 172.21.15.100:0/112386678 >> v2:172.21.15.100:3300/0 conn(0x55757da55060 0x55757daef0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:03.334 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036346a640 1 -- 172.21.15.100:0/112386678 >> v1:172.21.15.122:6789/0 conn(0x55757da539d0 legacy=0x55757da53db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:03.334 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036346a640 10 monclient: _finish_hunting 0 2024-09-13T11:53:03.334 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036346a640 1 monclient: found mon.noname-b 2024-09-13T11:53:03.334 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036346a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:03.334 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036346a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:53:03.334 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.320+0000 7f036346a640 1 -- 172.21.15.100:0/112386678 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55757d8bdf30 con 0x55757da544c0 2024-09-13T11:53:03.334 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.322+0000 7f036346a640 1 --2- 172.21.15.100:0/112386678 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55757da544c0 0x55757daf1630 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f035400a5a0 tx=0x7f0354045d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9418bc55778894ab server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:03.334 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.322+0000 7f0361206640 1 -- 172.21.15.100:0/112386678 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035405a070 con 0x55757da544c0 2024-09-13T11:53:03.334 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.322+0000 7f0361206640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:03.334 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.322+0000 7f0361206640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:03.334 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.322+0000 7f0361206640 10 monclient: dump: 2024-09-13T11:53:03.334 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:03.334 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:03.334 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:03.335 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:03.335 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:03.335 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:03.335 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:03.335 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:03.335 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:03.335 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:03.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.322+0000 7f0361206640 10 monclient: _finish_auth 0 2024-09-13T11:53:03.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.322+0000 7f0361206640 10 monclient: _check_auth_tickets 2024-09-13T11:53:03.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.322+0000 7f0361206640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:03.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.322+0000 7f0361206640 1 -- 172.21.15.100:0/112386678 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0354055640 con 0x55757da544c0 2024-09-13T11:53:03.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.322+0000 7f0361206640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:03.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.322+0000 7f0361206640 1 -- 172.21.15.100:0/112386678 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0354055a30 con 0x55757da544c0 2024-09-13T11:53:03.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.322+0000 7f0361206640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:03.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.322+0000 7f0361206640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:03.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.322+0000 7f0361206640 10 monclient: dump: 2024-09-13T11:53:03.336 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:03.336 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:03.336 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:03.336 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:03.336 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:03.336 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:03.336 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:03.336 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:03.336 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:03.336 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:03.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.322+0000 7f036371cf00 5 monclient: authenticate success, global_id 4202 2024-09-13T11:53:03.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.322+0000 7f036371cf00 10 monclient: get_monmap_and_config success 2024-09-13T11:53:03.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.322+0000 7f036371cf00 4 set_mon_vals no callback set 2024-09-13T11:53:03.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.323+0000 7f036371cf00 10 monclient: shutdown 2024-09-13T11:53:03.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.323+0000 7f036371cf00 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:03.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.323+0000 7f036371cf00 1 -- 172.21.15.100:0/112386678 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55757da544c0 msgr2=0x55757daf1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:03.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.323+0000 7f036371cf00 1 --2- 172.21.15.100:0/112386678 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55757da544c0 0x55757daf1630 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f035400a5a0 tx=0x7f0354045d60 comp rx=0 tx=0).stop 2024-09-13T11:53:03.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.323+0000 7f036371cf00 1 -- 172.21.15.100:0/112386678 shutdown_connections 2024-09-13T11:53:03.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.323+0000 7f036371cf00 1 --2- 172.21.15.100:0/112386678 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55757da544c0 0x55757daf1630 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:03.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.323+0000 7f036371cf00 1 --2- 172.21.15.100:0/112386678 >> v2:172.21.15.100:3300/0 conn(0x55757da55060 0x55757daef0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:03.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.323+0000 7f036371cf00 1 -- 172.21.15.100:0/112386678 >> 172.21.15.100:0/112386678 conn(0x55757da3ac40 msgr2=0x55757da3b050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:03.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.323+0000 7f036371cf00 1 -- 172.21.15.100:0/112386678 shutdown_connections 2024-09-13T11:53:03.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.323+0000 7f036371cf00 1 -- 172.21.15.100:0/112386678 wait complete. 2024-09-13T11:53:03.338 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.323+0000 7f036371cf00 10 monclient: build_initial_monmap 2024-09-13T11:53:03.338 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.323+0000 7f036371cf00 10 monclient: monmap: 2024-09-13T11:53:03.338 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:03.338 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:03.338 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:03.338 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:03.338 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:03.338 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:03.338 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:03.338 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:03.338 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:03.338 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:03.338 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.323+0000 7f036371cf00 1 Processor -- start 2024-09-13T11:53:03.338 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f036371cf00 1 -- start start 2024-09-13T11:53:03.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f036371cf00 10 monclient: init 2024-09-13T11:53:03.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f036371cf00 10 monclient: _reopen_session rank -1 2024-09-13T11:53:03.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f036371cf00 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:53:03.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f036371cf00 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55757da539d0 0x55757daf9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:03.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f036371cf00 10 monclient(hunting): picked mon.noname-a con 0x55757da539d0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:03.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f036371cf00 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55757da544c0 0x55757daf99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:03.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f036371cf00 10 monclient(hunting): picked mon.noname-c con 0x55757da544c0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:03.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f036371cf00 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55757da55060 0x55757dad48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:03.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f036371cf00 10 monclient(hunting): picked mon.noname-b con 0x55757da55060 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:03.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f036371cf00 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:03.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f036371cf00 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x55757d8cad90 con 0x55757da539d0 2024-09-13T11:53:03.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f036371cf00 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:03.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f036371cf00 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55757d927e00 con 0x55757da55060 2024-09-13T11:53:03.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f036371cf00 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:03.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f036371cf00 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x55757da586e0 con 0x55757da544c0 2024-09-13T11:53:03.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f036371cf00 10 monclient(hunting): _renew_subs 2024-09-13T11:53:03.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f036371cf00 10 monclient(hunting): authenticate will time out at 763.917507s 2024-09-13T11:53:03.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0362208640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55757da539d0 0x55757daf9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:03.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0361a07640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55757da544c0 0x55757daf99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:03.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0361a07640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55757da544c0 0x55757daf99a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47200/0 (socket says 172.21.15.100:47200) 2024-09-13T11:53:03.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0361a07640 1 -- 172.21.15.100:0/1040598066 learned_addr learned my addr 172.21.15.100:0/1040598066 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:03.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0361a07640 10 monclient(hunting): get_auth_request con 0x55757da544c0 auth_method 0 2024-09-13T11:53:03.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0361a07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:03.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0361a07640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:03.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0361a07640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:03.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0362208640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55757da539d0 0x55757daf9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49390/0 (socket says 172.21.15.100:49390) 2024-09-13T11:53:03.341 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0362208640 10 monclient(hunting): get_auth_request con 0x55757da539d0 auth_method 0 2024-09-13T11:53:03.341 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0362208640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:03.341 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0362208640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:03.341 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0362208640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:03.341 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0361a07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:03.341 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0361a07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:03.341 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0361a07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:03.341 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0362208640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:03.341 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0362208640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:03.341 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0362208640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:03.341 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f036346a640 1 --2- 172.21.15.100:0/1040598066 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55757da55060 0x55757dad48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:03.341 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f036346a640 10 monclient(hunting): get_auth_request con 0x55757da55060 auth_method 0 2024-09-13T11:53:03.341 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f036346a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:03.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f036346a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:03.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f036346a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:03.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0361a07640 10 monclient(hunting): handle_auth_done global_id 4205 payload 824 2024-09-13T11:53:03.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0361a07640 1 -- 172.21.15.100:0/1040598066 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55757da55060 msgr2=0x55757dad48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:03.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0361a07640 1 --2- 172.21.15.100:0/1040598066 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55757da55060 0x55757dad48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:03.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0361a07640 1 -- 172.21.15.100:0/1040598066 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55757da539d0 msgr2=0x55757daf9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:03.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0361a07640 1 --2- 172.21.15.100:0/1040598066 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55757da539d0 0x55757daf9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:03.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0361a07640 10 monclient: _finish_hunting 0 2024-09-13T11:53:03.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0361a07640 1 monclient: found mon.noname-c 2024-09-13T11:53:03.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0361a07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:03.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0361a07640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:53:03.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0361a07640 1 -- 172.21.15.100:0/1040598066 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55757d8a30c0 con 0x55757da544c0 2024-09-13T11:53:03.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.324+0000 7f0362208640 1 --2- 172.21.15.100:0/1040598066 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55757da539d0 0x55757daf9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:03.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.325+0000 7f0361a07640 1 --2- 172.21.15.100:0/1040598066 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55757da544c0 0x55757daf99a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f034c0087e0 tx=0x7f034c0088d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:03.343 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.325+0000 7f03527fc640 1 -- 172.21.15.100:0/1040598066 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c012790 con 0x55757da544c0 2024-09-13T11:53:03.343 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.325+0000 7f03527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:03.343 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.325+0000 7f03527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:03.343 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.325+0000 7f03527fc640 10 monclient: dump: 2024-09-13T11:53:03.343 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:03.343 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:03.343 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:03.343 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:03.343 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:03.343 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:03.343 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:03.343 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:03.343 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:03.343 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:03.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.325+0000 7f03527fc640 10 monclient: _finish_auth 0 2024-09-13T11:53:03.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.325+0000 7f03527fc640 10 monclient: _check_auth_tickets 2024-09-13T11:53:03.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.325+0000 7f03527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:03.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.325+0000 7f03527fc640 1 -- 172.21.15.100:0/1040598066 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f034c018070 con 0x55757da544c0 2024-09-13T11:53:03.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.325+0000 7f036371cf00 5 monclient: authenticate success, global_id 4205 2024-09-13T11:53:03.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.325+0000 7f036371cf00 10 monclient: _renew_subs 2024-09-13T11:53:03.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.325+0000 7f036371cf00 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:03.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.325+0000 7f036371cf00 1 -- 172.21.15.100:0/1040598066 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x55757dad8590 con 0x55757da544c0 2024-09-13T11:53:03.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.325+0000 7f03527fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:03.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.325+0000 7f03527fc640 1 -- 172.21.15.100:0/1040598066 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c012b30 con 0x55757da544c0 2024-09-13T11:53:03.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.325+0000 7f03527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:03.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.325+0000 7f03527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:03.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.325+0000 7f03527fc640 10 monclient: dump: 2024-09-13T11:53:03.345 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:03.345 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:03.345 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:03.345 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:03.345 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:03.345 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:03.345 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:03.345 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:03.345 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:03.345 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:03.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.325+0000 7f0360a05640 4 set_mon_vals no callback set 2024-09-13T11:53:03.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.325+0000 7f036371cf00 10 monclient: _renew_subs 2024-09-13T11:53:03.346 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.325+0000 7f036371cf00 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:03.346 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.325+0000 7f036371cf00 1 -- 172.21.15.100:0/1040598066 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x55757dad6230 con 0x55757da544c0 2024-09-13T11:53:03.346 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.325+0000 7f03527fc640 1 -- 172.21.15.100:0/1040598066 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f034c012cd0 con 0x55757da544c0 2024-09-13T11:53:03.346 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.326+0000 7f03527fc640 1 --2- 172.21.15.100:0/1040598066 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f034003c890 0x7f034003ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:03.346 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.326+0000 7f03527fc640 1 -- 172.21.15.100:0/1040598066 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4335+0+0 (secure 0 0 0) 0x7f034c051570 con 0x55757da544c0 2024-09-13T11:53:03.346 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.326+0000 7f036371cf00 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-13T11:53:03.346 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.326+0000 7f036371cf00 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-13T11:53:03.346 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.326+0000 7f036371cf00 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:03.346 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.326+0000 7f036371cf00 1 -- 172.21.15.100:0/1040598066 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x55757dad6d90 con 0x55757da544c0 2024-09-13T11:53:03.346 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.326+0000 7f0362208640 1 --2- 172.21.15.100:0/1040598066 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f034003c890 0x7f034003ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:03.347 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.326+0000 7f0362208640 10 monclient: get_auth_request con 0x7f034003c890 auth_method 0 2024-09-13T11:53:03.347 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:03.327+0000 7f0362208640 1 --2- 172.21.15.100:0/1040598066 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f034003c890 0x7f034003ed50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0358005fd0 tx=0x7f0358007040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:04.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:04.255+0000 7f03527fc640 1 -- 172.21.15.100:0/1040598066 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v16) ==== 141+0+0 (secure 0 0 0) 0x7f034c013dc0 con 0x55757da544c0 2024-09-13T11:53:04.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:04.255+0000 7f03527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-13T11:53:04.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:04.255+0000 7f03527fc640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-13T11:53:04.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:04.256+0000 7f036371cf00 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:04.259 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:04.256+0000 7f036371cf00 1 -- 172.21.15.100:0/1040598066 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0354055640 con 0x55757da544c0 2024-09-13T11:53:04.272 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:04.270+0000 7f03527fc640 1 -- 172.21.15.100:0/1040598066 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f034c01c410 con 0x55757da544c0 2024-09-13T11:53:04.273 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:04.270+0000 7f03527fc640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-13T11:53:04.273 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:04.270+0000 7f0360a05640 10 monclient: _renew_subs 2024-09-13T11:53:04.273 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:04.270+0000 7f0360a05640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:04.273 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:04.270+0000 7f0360a05640 1 -- 172.21.15.100:0/1040598066 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=16}) -- 0x7f0344002ae0 con 0x55757da544c0 2024-09-13T11:53:04.273 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:04.270+0000 7f03527fc640 1 -- 172.21.15.100:0/1040598066 <== mon.2 v2:172.21.15.100:3301/0 8 ==== osd_map(16..16 src has 1..16) ==== 658+0+0 (secure 0 0 0) 0x7f034c051240 con 0x55757da544c0 2024-09-13T11:53:04.273 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:04.270+0000 7f036371cf00 1 --2- 172.21.15.100:0/1040598066 >> [v2:172.21.15.122:6800/904725860,v1:172.21.15.122:6801/904725860] conn(0x55757dad5920 0x55757db02420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:04.273 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:04.271+0000 7f036371cf00 1 -- 172.21.15.100:0/1040598066 --> [v2:172.21.15.122:6800/904725860,v1:172.21.15.122:6801/904725860] -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16) -- 0x55757db02960 con 0x55757dad5920 2024-09-13T11:53:04.273 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:04.271+0000 7f036346a640 1 --2- 172.21.15.100:0/1040598066 >> [v2:172.21.15.122:6800/904725860,v1:172.21.15.122:6801/904725860] conn(0x55757dad5920 0x55757db02420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:04.273 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:04.271+0000 7f036346a640 10 monclient: get_auth_request con 0x55757dad5920 auth_method 0 2024-09-13T11:53:04.274 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:04.271+0000 7f036346a640 1 --2- 172.21.15.100:0/1040598066 >> [v2:172.21.15.122:6800/904725860,v1:172.21.15.122:6801/904725860] conn(0x55757dad5920 0x55757db02420 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:04.274 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:04.273+0000 7f036346a640 1 -- 172.21.15.100:0/1040598066 <== osd.5 v2:172.21.15.122:6800/904725860 1 ==== osd_op_reply(1 rbd_trash [create] v16'1 uv1 ondisk = 0) ==== 153+0+0 (crc 0 0 0) 0x7f03540558f0 con 0x55757dad5920 2024-09-13T11:53:04.276 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:04.275+0000 7f036371cf00 1 -- 172.21.15.100:0/1040598066 --> [v2:172.21.15.122:6800/904725860,v1:172.21.15.122:6801/904725860] -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16) -- 0x55757db552a0 con 0x55757dad5920 2024-09-13T11:53:04.277 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:04.276+0000 7f036346a640 1 -- 172.21.15.100:0/1040598066 <== osd.5 v2:172.21.15.122:6800/904725860 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f03540558f0 con 0x55757dad5920 2024-09-13T11:53:04.277 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:04.276+0000 7f036371cf00 1 -- 172.21.15.100:0/1040598066 --> [v2:172.21.15.122:6800/904725860,v1:172.21.15.122:6801/904725860] -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16) -- 0x55757db54380 con 0x55757dad5920 2024-09-13T11:53:04.278 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:04.276+0000 7f036346a640 1 -- 172.21.15.100:0/1040598066 <== osd.5 v2:172.21.15.122:6800/904725860 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f03540558f0 con 0x55757dad5920 2024-09-13T11:53:04.278 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:04.276+0000 7f0353fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:04.278 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:04.276+0000 7f0353fff640 1 -- 172.21.15.100:0/1040598066 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7f0338001f00 con 0x55757da544c0 2024-09-13T11:53:05.259 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:05.257+0000 7f03527fc640 1 -- 172.21.15.100:0/1040598066 <== mon.2 v2:172.21.15.100:3301/0 9 ==== pool_op_reply(tid 4 (0) Success v17) ==== 55+0+0 (secure 0 0 0) 0x7f034c04c030 con 0x55757da544c0 2024-09-13T11:53:05.259 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:05.257+0000 7f03527fc640 10 monclient: _renew_subs 2024-09-13T11:53:05.259 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:05.257+0000 7f03527fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:05.259 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:05.257+0000 7f03527fc640 1 -- 172.21.15.100:0/1040598066 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=17}) -- 0x7f0340042af0 con 0x55757da544c0 2024-09-13T11:53:05.260 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:05.257+0000 7f03527fc640 1 -- 172.21.15.100:0/1040598066 <== mon.2 v2:172.21.15.100:3301/0 10 ==== osd_map(17..17 src has 1..17) ==== 626+0+0 (secure 0 0 0) 0x7f034c01a310 con 0x55757da544c0 2024-09-13T11:53:05.260 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:05.257+0000 7f0360a05640 1 -- 172.21.15.100:0/1040598066 --> [v2:172.21.15.122:6800/904725860,v1:172.21.15.122:6801/904725860] -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) -- 0x7f0344005790 con 0x55757dad5920 2024-09-13T11:53:05.262 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:05.261+0000 7f036346a640 1 -- 172.21.15.100:0/1040598066 <== osd.5 v2:172.21.15.122:6800/904725860 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v17'1 uv1 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f03540558f0 con 0x55757dad5920 2024-09-13T11:53:05.262 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:05.261+0000 7f0353fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:05.263 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:05.261+0000 7f0353fff640 1 -- 172.21.15.100:0/1040598066 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v17) -- 0x7f0338003350 con 0x55757da544c0 2024-09-13T11:53:06.267 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.266+0000 7f03527fc640 1 -- 172.21.15.100:0/1040598066 <== mon.2 v2:172.21.15.100:3301/0 11 ==== pool_op_reply(tid 6 (0) Success v18) ==== 43+0+0 (secure 0 0 0) 0x7f034c02b070 con 0x55757da544c0 2024-09-13T11:53:06.268 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.266+0000 7f03527fc640 10 monclient: _renew_subs 2024-09-13T11:53:06.268 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.266+0000 7f03527fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:06.268 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.266+0000 7f03527fc640 1 -- 172.21.15.100:0/1040598066 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=18}) -- 0x7f0340042d70 con 0x55757da544c0 2024-09-13T11:53:06.268 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.266+0000 7f03527fc640 1 -- 172.21.15.100:0/1040598066 <== mon.2 v2:172.21.15.100:3301/0 12 ==== osd_map(18..18 src has 1..18) ==== 654+0+0 (secure 0 0 0) 0x7f034c01a6a0 con 0x55757da544c0 2024-09-13T11:53:06.268 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.266+0000 7f0360a05640 1 -- 172.21.15.100:0/1040598066 --> [v2:172.21.15.122:6800/904725860,v1:172.21.15.122:6801/904725860] -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) -- 0x7f0344005b50 con 0x55757dad5920 2024-09-13T11:53:06.274 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.272+0000 7f036346a640 1 -- 172.21.15.100:0/1040598066 <== osd.5 v2:172.21.15.122:6800/904725860 5 ==== osd_op_reply(7 rbd_info [write 0~19] v18'2 uv2 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f03540558f0 con 0x55757dad5920 2024-09-13T11:53:06.274 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.272+0000 7f036371cf00 1 -- 172.21.15.100:0/1040598066 >> [v2:172.21.15.122:6800/904725860,v1:172.21.15.122:6801/904725860] conn(0x55757dad5920 msgr2=0x55757db02420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:06.274 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.272+0000 7f036371cf00 1 --2- 172.21.15.100:0/1040598066 >> [v2:172.21.15.122:6800/904725860,v1:172.21.15.122:6801/904725860] conn(0x55757dad5920 0x55757db02420 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:06.274 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.272+0000 7f036371cf00 1 -- 172.21.15.100:0/1040598066 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f034003c890 msgr2=0x7f034003ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:06.274 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.272+0000 7f036371cf00 1 --2- 172.21.15.100:0/1040598066 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f034003c890 0x7f034003ed50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0358005fd0 tx=0x7f0358007040 comp rx=0 tx=0).stop 2024-09-13T11:53:06.274 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.272+0000 7f036371cf00 10 monclient: shutdown 2024-09-13T11:53:06.275 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.272+0000 7f036371cf00 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:06.275 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.272+0000 7f036371cf00 1 -- 172.21.15.100:0/1040598066 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55757da544c0 msgr2=0x55757daf99a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:06.275 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.272+0000 7f036371cf00 1 --2- 172.21.15.100:0/1040598066 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55757da544c0 0x55757daf99a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f034c0087e0 tx=0x7f034c0088d0 comp rx=0 tx=0).stop 2024-09-13T11:53:06.275 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.272+0000 7f036346a640 1 -- 172.21.15.100:0/1040598066 reap_dead start 2024-09-13T11:53:06.275 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.272+0000 7f036371cf00 1 -- 172.21.15.100:0/1040598066 shutdown_connections 2024-09-13T11:53:06.275 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.272+0000 7f036371cf00 1 -- 172.21.15.100:0/1040598066 >> 172.21.15.100:0/1040598066 conn(0x55757da3ac40 msgr2=0x55757da4d730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:06.275 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.273+0000 7f036371cf00 1 -- 172.21.15.100:0/1040598066 shutdown_connections 2024-09-13T11:53:06.275 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.273+0000 7f036371cf00 1 -- 172.21.15.100:0/1040598066 wait complete. 2024-09-13T11:53:06.278 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-13T11:53:06.279 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-13T11:53:06.279 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-13T11:53:06.388 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.386+0000 7f2a41317640 10 monclient: get_monmap_and_config 2024-09-13T11:53:06.388 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.386+0000 7f2a41317640 10 monclient: build_initial_monmap 2024-09-13T11:53:06.388 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.386+0000 7f2a41317640 10 monclient: monmap: 2024-09-13T11:53:06.388 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:06.388 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:06.388 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:06.387507+0000 2024-09-13T11:53:06.388 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:06.387507+0000 2024-09-13T11:53:06.389 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:06.389 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:06.389 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:06.389 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:06.389 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:06.389 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:06.389 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:06.389 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:06.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.387+0000 7f2a41317640 1 Processor -- start 2024-09-13T11:53:06.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.387+0000 7f2a41317640 1 -- start start 2024-09-13T11:53:06.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.387+0000 7f2a41317640 10 monclient: init 2024-09-13T11:53:06.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.387+0000 7f2a41317640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:06.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.387+0000 7f2a41317640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-13T11:53:06.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.387+0000 7f2a41317640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f2a3c137a60 0x7f2a3c0a2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:06.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a41317640 10 monclient(hunting): picked mon.noname-a con 0x7f2a3c137a60 addr v2:172.21.15.100:3300/0 2024-09-13T11:53:06.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a41317640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2a3c137e40 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:06.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a41317640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2a3c138250 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:06.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a41317640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a3c0a5770 con 0x7f2a3c137e40 2024-09-13T11:53:06.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a41317640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a3c0a5be0 con 0x7f2a3c138250 2024-09-13T11:53:06.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a41317640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:06.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a41317640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f2a3c0a4410 con 0x7f2a3c137a60 2024-09-13T11:53:06.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a41317640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:06.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a41317640 10 monclient(hunting): authenticate will time out at 766.981113s 2024-09-13T11:53:06.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a3affd640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f2a3c137a60 0x7f2a3c0a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:06.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a33fff640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f2a3c137e40 0x7f2a3c0a4950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:57860/0 (socket says 172.21.15.100:57860) 2024-09-13T11:53:06.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a33fff640 1 -- 172.21.15.100:0/3422212283 learned_addr learned my addr 172.21.15.100:0/3422212283 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:53:06.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a3affd640 10 monclient(hunting): get_auth_request con 0x7f2a3c137a60 auth_method 0 2024-09-13T11:53:06.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:06.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a3affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:06.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:06.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:06.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:06.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:06.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a3affd640 10 monclient(hunting): handle_auth_done global_id 4233 payload 274 2024-09-13T11:53:06.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a3affd640 1 -- 172.21.15.100:0/3422212283 >> v1:172.21.15.122:6789/0 conn(0x7f2a3c138250 legacy=0x7f2a3c0a5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:06.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a3affd640 1 -- 172.21.15.100:0/3422212283 >> v1:172.21.15.100:6789/0 conn(0x7f2a3c137e40 legacy=0x7f2a3c0a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:06.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a3affd640 10 monclient: _finish_hunting 0 2024-09-13T11:53:06.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a3affd640 1 monclient: found mon.noname-a 2024-09-13T11:53:06.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a3a7fc640 1 -- 172.21.15.100:0/3422212283 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 457416118 0 0) 0x7f2a3c0a5770 con 0x7f2a3c137e40 2024-09-13T11:53:06.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:06.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a3affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:06.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a3affd640 1 -- 172.21.15.100:0/3422212283 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a3c0a76b0 con 0x7f2a3c137a60 2024-09-13T11:53:06.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.388+0000 7f2a3a7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:53:06.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.389+0000 7f2a3affd640 1 --2- 172.21.15.100:0/3422212283 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a3c137a60 0x7f2a3c0a2980 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c0a5db0 tx=0x7f2a2c02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e14802787817d7e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:06.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.389+0000 7f2a3a7fc640 1 -- 172.21.15.100:0/3422212283 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c037070 con 0x7f2a3c137a60 2024-09-13T11:53:06.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.389+0000 7f2a3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:06.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.389+0000 7f2a3a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:06.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.389+0000 7f2a3a7fc640 10 monclient: dump: 2024-09-13T11:53:06.401 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:06.401 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:06.401 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:06.401 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:06.401 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:06.401 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:06.401 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:06.401 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:06.402 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:06.402 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:06.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.389+0000 7f2a3a7fc640 10 monclient: _finish_auth 0 2024-09-13T11:53:06.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.389+0000 7f2a3a7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:53:06.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.389+0000 7f2a3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:06.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.389+0000 7f2a3a7fc640 1 -- 172.21.15.100:0/3422212283 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a2c02ae50 con 0x7f2a3c137a60 2024-09-13T11:53:06.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.389+0000 7f2a3a7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:06.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.389+0000 7f2a41317640 5 monclient: authenticate success, global_id 4233 2024-09-13T11:53:06.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.389+0000 7f2a41317640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:06.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.389+0000 7f2a41317640 4 set_mon_vals no callback set 2024-09-13T11:53:06.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.389+0000 7f2a3a7fc640 1 -- 172.21.15.100:0/3422212283 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c032660 con 0x7f2a3c137a60 2024-09-13T11:53:06.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.389+0000 7f2a3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:06.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.389+0000 7f2a3a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:06.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.389+0000 7f2a3a7fc640 10 monclient: dump: 2024-09-13T11:53:06.403 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:06.403 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:06.403 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:06.403 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:06.403 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:06.403 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:06.403 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:06.403 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:06.403 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:06.403 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:06.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.389+0000 7f2a41317640 10 monclient: shutdown 2024-09-13T11:53:06.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.389+0000 7f2a41317640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:06.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.389+0000 7f2a41317640 1 -- 172.21.15.100:0/3422212283 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a3c137a60 msgr2=0x7f2a3c0a2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:06.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.389+0000 7f2a41317640 1 --2- 172.21.15.100:0/3422212283 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a3c137a60 0x7f2a3c0a2980 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c0a5db0 tx=0x7f2a2c02a100 comp rx=0 tx=0).stop 2024-09-13T11:53:06.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.390+0000 7f2a41317640 1 -- 172.21.15.100:0/3422212283 shutdown_connections 2024-09-13T11:53:06.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.390+0000 7f2a41317640 1 --2- 172.21.15.100:0/3422212283 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a3c137a60 0x7f2a3c0a2980 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:06.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.390+0000 7f2a41317640 1 -- 172.21.15.100:0/3422212283 >> 172.21.15.100:0/3422212283 conn(0x7f2a3c0aa580 msgr2=0x7f2a3c0a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:06.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.390+0000 7f2a41317640 1 -- 172.21.15.100:0/3422212283 shutdown_connections 2024-09-13T11:53:06.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.390+0000 7f2a41317640 1 -- 172.21.15.100:0/3422212283 wait complete. 2024-09-13T11:53:06.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.390+0000 7f2a41317640 10 monclient: build_initial_monmap 2024-09-13T11:53:06.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.390+0000 7f2a41317640 10 monclient: monmap: 2024-09-13T11:53:06.404 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:06.405 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:06.405 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:06.405 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:06.405 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:06.405 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:06.405 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:06.405 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:06.405 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:06.405 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:06.405 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.390+0000 7f2a41317640 1 Processor -- start 2024-09-13T11:53:06.405 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a41317640 1 -- start start 2024-09-13T11:53:06.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a41317640 10 monclient: init 2024-09-13T11:53:06.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a41317640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:06.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a41317640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:53:06.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a41317640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a3c137a60 0x7f2a3c13c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:06.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a41317640 10 monclient(hunting): picked mon.noname-a con 0x7f2a3c137a60 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:06.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a41317640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a3c137e40 0x7f2a3c13c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:06.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a41317640 10 monclient(hunting): picked mon.noname-b con 0x7f2a3c137e40 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:06.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a41317640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a3c138250 0x7f2a3c13cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:06.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a41317640 10 monclient(hunting): picked mon.noname-c con 0x7f2a3c138250 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:06.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a41317640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:06.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a41317640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c06fd80 con 0x7f2a3c137a60 2024-09-13T11:53:06.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a41317640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:06.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a41317640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c0a1700 con 0x7f2a3c137e40 2024-09-13T11:53:06.407 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a41317640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:06.407 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a41317640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f2a3c06ff00 con 0x7f2a3c138250 2024-09-13T11:53:06.407 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a41317640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:06.407 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a41317640 10 monclient(hunting): authenticate will time out at 766.984555s 2024-09-13T11:53:06.407 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3affd640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a3c137a60 0x7f2a3c13c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:06.407 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3affd640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a3c137a60 0x7f2a3c13c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49404/0 (socket says 172.21.15.100:49404) 2024-09-13T11:53:06.407 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3affd640 1 -- 172.21.15.100:0/3695462226 learned_addr learned my addr 172.21.15.100:0/3695462226 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:06.407 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3affd640 10 monclient(hunting): get_auth_request con 0x7f2a3c137a60 auth_method 0 2024-09-13T11:53:06.407 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:06.407 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:06.407 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:06.407 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3b7fe640 1 --2- 172.21.15.100:0/3695462226 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a3c138250 0x7f2a3c13cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:06.408 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:06.408 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:06.408 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:06.408 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3b7fe640 10 monclient(hunting): get_auth_request con 0x7f2a3c138250 auth_method 0 2024-09-13T11:53:06.408 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:06.408 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:06.408 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:06.408 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:06.408 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:06.408 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:06.409 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a33fff640 1 --2- 172.21.15.100:0/3695462226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a3c137e40 0x7f2a3c13c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:06.409 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3affd640 10 monclient(hunting): handle_auth_done global_id 4239 payload 824 2024-09-13T11:53:06.409 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3affd640 1 -- 172.21.15.100:0/3695462226 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a3c138250 msgr2=0x7f2a3c13cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:06.409 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3affd640 1 --2- 172.21.15.100:0/3695462226 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a3c138250 0x7f2a3c13cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:06.409 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3affd640 1 -- 172.21.15.100:0/3695462226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a3c137e40 msgr2=0x7f2a3c13c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:06.409 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3affd640 1 --2- 172.21.15.100:0/3695462226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a3c137e40 0x7f2a3c13c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:06.409 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3affd640 10 monclient: _finish_hunting 0 2024-09-13T11:53:06.409 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3affd640 1 monclient: found mon.noname-a 2024-09-13T11:53:06.409 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:06.409 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:06.409 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a3affd640 1 -- 172.21.15.100:0/3695462226 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a3c13fe60 con 0x7f2a3c137a60 2024-09-13T11:53:06.409 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a33fff640 10 monclient: get_auth_request con 0x7f2a3c137e40 auth_method 0 2024-09-13T11:53:06.410 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.391+0000 7f2a33fff640 1 --2- 172.21.15.100:0/3695462226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a3c137e40 0x7f2a3c13c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:53:06.410 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.392+0000 7f2a3affd640 1 --2- 172.21.15.100:0/3695462226 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a3c137a60 0x7f2a3c13c360 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c000c00 tx=0x7f2a2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:06.410 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.392+0000 7f2a337fe640 1 -- 172.21.15.100:0/3695462226 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c037040 con 0x7f2a3c137a60 2024-09-13T11:53:06.410 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.392+0000 7f2a337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:06.410 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.392+0000 7f2a337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:06.410 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.392+0000 7f2a337fe640 10 monclient: dump: 2024-09-13T11:53:06.410 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:06.410 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:06.410 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:06.410 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:06.410 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:06.410 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:06.411 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:06.411 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:06.411 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:06.411 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:06.411 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.392+0000 7f2a337fe640 10 monclient: _finish_auth 0 2024-09-13T11:53:06.411 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.392+0000 7f2a337fe640 10 monclient: _check_auth_tickets 2024-09-13T11:53:06.411 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.392+0000 7f2a337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:06.411 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.392+0000 7f2a41317640 5 monclient: authenticate success, global_id 4239 2024-09-13T11:53:06.411 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.392+0000 7f2a337fe640 1 -- 172.21.15.100:0/3695462226 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a2c004230 con 0x7f2a3c137a60 2024-09-13T11:53:06.411 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.392+0000 7f2a337fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:06.411 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.392+0000 7f2a337fe640 1 -- 172.21.15.100:0/3695462226 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c003610 con 0x7f2a3c137a60 2024-09-13T11:53:06.411 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.392+0000 7f2a41317640 10 monclient: _renew_subs 2024-09-13T11:53:06.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.392+0000 7f2a41317640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:06.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.392+0000 7f2a41317640 1 -- 172.21.15.100:0/3695462226 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a3c140140 con 0x7f2a3c137a60 2024-09-13T11:53:06.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.392+0000 7f2a337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:06.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.392+0000 7f2a337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:06.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.392+0000 7f2a337fe640 10 monclient: dump: 2024-09-13T11:53:06.412 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:06.412 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:06.412 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:06.412 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:06.412 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:06.412 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:06.412 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:06.412 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:06.412 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:06.413 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:06.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.392+0000 7f2a39ffb640 4 set_mon_vals no callback set 2024-09-13T11:53:06.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.392+0000 7f2a41317640 10 monclient: _renew_subs 2024-09-13T11:53:06.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.392+0000 7f2a41317640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:06.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.392+0000 7f2a41317640 1 -- 172.21.15.100:0/3695462226 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a3c0a8780 con 0x7f2a3c137a60 2024-09-13T11:53:06.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.393+0000 7f2a337fe640 1 -- 172.21.15.100:0/3695462226 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f2a2c0037b0 con 0x7f2a3c137a60 2024-09-13T11:53:06.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.393+0000 7f2a41317640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:06.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.393+0000 7f2a41317640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:06.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.393+0000 7f2a41317640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:06.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.393+0000 7f2a41317640 1 -- 172.21.15.100:0/3695462226 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a3c0a2980 con 0x7f2a3c137a60 2024-09-13T11:53:06.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.393+0000 7f2a337fe640 1 --2- 172.21.15.100:0/3695462226 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f2a1c03dfd0 0x7f2a1c040490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:06.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.393+0000 7f2a337fe640 1 -- 172.21.15.100:0/3695462226 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7f2a2c0734a0 con 0x7f2a3c137a60 2024-09-13T11:53:06.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.393+0000 7f2a33fff640 1 --2- 172.21.15.100:0/3695462226 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f2a1c03dfd0 0x7f2a1c040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:06.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.393+0000 7f2a33fff640 10 monclient: get_auth_request con 0x7f2a1c03dfd0 auth_method 0 2024-09-13T11:53:06.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.394+0000 7f2a33fff640 1 --2- 172.21.15.100:0/3695462226 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f2a1c03dfd0 0x7f2a1c040490 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2a24007fd0 tx=0x7f2a24009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:06.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.398+0000 7f2a337fe640 1 -- 172.21.15.100:0/3695462226 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f2a2c03c3d0 con 0x7f2a3c137a60 2024-09-13T11:53:06.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.398+0000 7f2a337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:06.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.398+0000 7f2a337fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:06.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.573+0000 7f2a41317640 10 start_mon_command cmd=[{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-13T11:53:06.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.573+0000 7f2a41317640 10 monclient: _send_command 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-13T11:53:06.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.573+0000 7f2a41317640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:06.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.573+0000 7f2a41317640 1 -- 172.21.15.100:0/3695462226 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f2a3c06a0f0 con 0x7f2a3c137a60 2024-09-13T11:53:06.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.573+0000 7f2a337fe640 1 -- 172.21.15.100:0/3695462226 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v6) ==== 86+0+159 (secure 0 0 0) 0x7f2a2c03ddd0 con 0x7f2a3c137a60 2024-09-13T11:53:06.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.573+0000 7f2a337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-13T11:53:06.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.573+0000 7f2a337fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:06.577 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:53:06.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.575+0000 7f2a41317640 1 -- 172.21.15.100:0/3695462226 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f2a1c03dfd0 msgr2=0x7f2a1c040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:06.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.575+0000 7f2a41317640 1 --2- 172.21.15.100:0/3695462226 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f2a1c03dfd0 0x7f2a1c040490 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2a24007fd0 tx=0x7f2a24009040 comp rx=0 tx=0).stop 2024-09-13T11:53:06.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.575+0000 7f2a41317640 10 monclient: shutdown 2024-09-13T11:53:06.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.575+0000 7f2a41317640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:06.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.575+0000 7f2a41317640 1 -- 172.21.15.100:0/3695462226 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a3c137a60 msgr2=0x7f2a3c13c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:06.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.575+0000 7f2a41317640 1 --2- 172.21.15.100:0/3695462226 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a3c137a60 0x7f2a3c13c360 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c000c00 tx=0x7f2a2c0046c0 comp rx=0 tx=0).stop 2024-09-13T11:53:06.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.575+0000 7f2a41317640 1 -- 172.21.15.100:0/3695462226 shutdown_connections 2024-09-13T11:53:06.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.575+0000 7f2a41317640 1 --2- 172.21.15.100:0/3695462226 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f2a1c03dfd0 0x7f2a1c040490 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:06.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.575+0000 7f2a41317640 1 --2- 172.21.15.100:0/3695462226 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a3c138250 0x7f2a3c13cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:06.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.575+0000 7f2a41317640 1 --2- 172.21.15.100:0/3695462226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a3c137e40 0x7f2a3c13c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:06.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.575+0000 7f2a41317640 1 --2- 172.21.15.100:0/3695462226 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a3c137a60 0x7f2a3c13c360 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:06.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.575+0000 7f2a41317640 1 -- 172.21.15.100:0/3695462226 >> 172.21.15.100:0/3695462226 conn(0x7f2a3c0aa580 msgr2=0x7f2a3c0a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:06.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.575+0000 7f2a41317640 1 -- 172.21.15.100:0/3695462226 shutdown_connections 2024-09-13T11:53:06.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.576+0000 7f2a41317640 1 -- 172.21.15.100:0/3695462226 wait complete. 2024-09-13T11:53:06.588 INFO:teuthology.orchestra.run.smithi100.stdout:[{"version":6,"timestamp":"2024-09-13T11:52:55.980020+0000","name":"","changes":[{"name":"osd.4/osd_mclock_max_capacity_iops_ssd","new_value":"6898.150398"}]}] 2024-09-13T11:53:06.588 INFO:tasks.ceph_manager:config epoch is 6 2024-09-13T11:53:06.588 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T11:53:06.589 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-13T11:53:06.589 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-13T11:53:06.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.683+0000 7fbbbbdd0640 10 monclient: get_monmap_and_config 2024-09-13T11:53:06.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.683+0000 7fbbbbdd0640 10 monclient: build_initial_monmap 2024-09-13T11:53:06.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.683+0000 7fbbbbdd0640 10 monclient: monmap: 2024-09-13T11:53:06.686 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:06.686 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:06.686 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:06.684263+0000 2024-09-13T11:53:06.686 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:06.684263+0000 2024-09-13T11:53:06.686 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:06.687 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:06.687 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:06.687 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:06.687 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:06.687 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:06.688 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:06.688 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:06.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.683+0000 7fbbbbdd0640 1 Processor -- start 2024-09-13T11:53:06.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbbbdd0640 1 -- start start 2024-09-13T11:53:06.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbbbdd0640 10 monclient: init 2024-09-13T11:53:06.700 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbbbdd0640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:06.700 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbbbdd0640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-13T11:53:06.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbbbdd0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbbb40a37d0 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:06.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbbbdd0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbbb409fd40 0x7fbbb413cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:06.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbbbdd0640 10 monclient(hunting): picked mon.noname-c con 0x7fbbb409fd40 addr v2:172.21.15.122:3300/0 2024-09-13T11:53:06.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbbbdd0640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbb40a42c0 0x7fbbb413f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:06.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbbbdd0640 10 monclient(hunting): picked mon.noname-b con 0x7fbbb40a42c0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:06.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbbbdd0640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbb406fe10 con 0x7fbbb40a37d0 2024-09-13T11:53:06.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbbbdd0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:06.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbbbdd0640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7fbbb409ee80 con 0x7fbbb409fd40 2024-09-13T11:53:06.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbbbdd0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:06.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbbbdd0640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fbbb406ec20 con 0x7fbbb40a42c0 2024-09-13T11:53:06.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbbbdd0640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:06.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbbbdd0640 10 monclient(hunting): authenticate will time out at 767.277678s 2024-09-13T11:53:06.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbba36d640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbb40a42c0 0x7fbbb413f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:06.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbb9b6c640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fbbb40a37d0 0x7fbbb40a3bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:57864/0 (socket says 172.21.15.100:57864) 2024-09-13T11:53:06.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbb9b6c640 1 -- 172.21.15.100:0/2147831131 learned_addr learned my addr 172.21.15.100:0/2147831131 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:53:06.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbba36d640 10 monclient(hunting): get_auth_request con 0x7fbbb40a42c0 auth_method 0 2024-09-13T11:53:06.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbba36d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:06.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbba36d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:06.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbba36d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:06.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbba36d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:06.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbba36d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:06.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.684+0000 7fbbba36d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:06.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.685+0000 7fbbb8b6a640 1 -- 172.21.15.100:0/2147831131 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3347823859 0 0) 0x7fbbb406fe10 con 0x7fbbb40a37d0 2024-09-13T11:53:06.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.685+0000 7fbbb8b6a640 10 monclient(hunting): _negotiate 2024-09-13T11:53:06.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.685+0000 7fbbb8b6a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:06.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.685+0000 7fbbb8b6a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:06.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.685+0000 7fbbb8b6a640 10 monclient(hunting): my global_id is 4242 2024-09-13T11:53:06.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.685+0000 7fbbb8b6a640 1 -- 172.21.15.100:0/2147831131 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbba0003520 con 0x7fbbb40a37d0 2024-09-13T11:53:06.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.685+0000 7fbbb936b640 1 --2- 172.21.15.100:0/2147831131 >> v2:172.21.15.122:3300/0 conn(0x7fbbb409fd40 0x7fbbb413cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:06.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.685+0000 7fbbba36d640 10 monclient(hunting): handle_auth_done global_id 4211 payload 274 2024-09-13T11:53:06.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.685+0000 7fbbba36d640 1 -- 172.21.15.100:0/2147831131 >> v2:172.21.15.122:3300/0 conn(0x7fbbb409fd40 msgr2=0x7fbbb413cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:06.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.685+0000 7fbbba36d640 1 --2- 172.21.15.100:0/2147831131 >> v2:172.21.15.122:3300/0 conn(0x7fbbb409fd40 0x7fbbb413cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:06.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.685+0000 7fbbba36d640 1 -- 172.21.15.100:0/2147831131 >> v1:172.21.15.100:6789/0 conn(0x7fbbb40a37d0 legacy=0x7fbbb40a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:06.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.685+0000 7fbbba36d640 10 monclient: _finish_hunting 0 2024-09-13T11:53:06.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.685+0000 7fbbba36d640 1 monclient: found mon.noname-b 2024-09-13T11:53:06.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.685+0000 7fbbba36d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:06.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.685+0000 7fbbba36d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:53:06.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.685+0000 7fbbba36d640 1 -- 172.21.15.100:0/2147831131 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb40708d0 con 0x7fbbb40a42c0 2024-09-13T11:53:06.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.685+0000 7fbbb936b640 10 monclient: get_auth_request con 0x7fbbb409fd40 auth_method 0 2024-09-13T11:53:06.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.685+0000 7fbbb936b640 1 --2- 172.21.15.100:0/2147831131 >> v2:172.21.15.122:3300/0 conn(0x7fbbb409fd40 0x7fbbb413cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:53:06.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.685+0000 7fbbba36d640 1 --2- 172.21.15.100:0/2147831131 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbb40a42c0 0x7fbbb413f100 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbbac009d90 tx=0x7fbbac02a520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f15e9bdd2f301e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:06.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.685+0000 7fbbb8b6a640 1 -- 172.21.15.100:0/2147831131 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac007740 con 0x7fbbb40a42c0 2024-09-13T11:53:06.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.685+0000 7fbbb8b6a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:06.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.686+0000 7fbbb8b6a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:06.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.686+0000 7fbbb8b6a640 10 monclient: dump: 2024-09-13T11:53:06.704 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:06.704 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:06.704 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:06.704 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:06.704 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:06.704 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:06.705 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:06.705 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:06.705 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:06.705 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:06.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.686+0000 7fbbb8b6a640 10 monclient: _finish_auth 0 2024-09-13T11:53:06.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.686+0000 7fbbb8b6a640 10 monclient: _check_auth_tickets 2024-09-13T11:53:06.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.686+0000 7fbbb8b6a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:06.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.686+0000 7fbbbbdd0640 5 monclient: authenticate success, global_id 4211 2024-09-13T11:53:06.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.686+0000 7fbbb8b6a640 1 -- 172.21.15.100:0/2147831131 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbac0078e0 con 0x7fbbb40a42c0 2024-09-13T11:53:06.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.686+0000 7fbbb8b6a640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:06.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.686+0000 7fbbbbdd0640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:06.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.686+0000 7fbbb8b6a640 1 -- 172.21.15.100:0/2147831131 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac007cb0 con 0x7fbbb40a42c0 2024-09-13T11:53:06.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.686+0000 7fbbbbdd0640 4 set_mon_vals no callback set 2024-09-13T11:53:06.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.686+0000 7fbbb8b6a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:06.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.686+0000 7fbbb8b6a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:06.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.686+0000 7fbbb8b6a640 10 monclient: dump: 2024-09-13T11:53:06.706 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:06.706 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:06.706 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:06.706 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:06.706 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:06.706 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:06.706 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:06.706 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:06.706 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:06.706 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:06.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.686+0000 7fbbbbdd0640 10 monclient: shutdown 2024-09-13T11:53:06.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.686+0000 7fbbbbdd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:06.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.686+0000 7fbbbbdd0640 1 -- 172.21.15.100:0/2147831131 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbb40a42c0 msgr2=0x7fbbb413f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:06.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.686+0000 7fbbbbdd0640 1 --2- 172.21.15.100:0/2147831131 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbb40a42c0 0x7fbbb413f100 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbbac009d90 tx=0x7fbbac02a520 comp rx=0 tx=0).stop 2024-09-13T11:53:06.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.686+0000 7fbbbbdd0640 1 -- 172.21.15.100:0/2147831131 shutdown_connections 2024-09-13T11:53:06.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.686+0000 7fbbbbdd0640 1 --2- 172.21.15.100:0/2147831131 >> v2:172.21.15.122:3300/0 conn(0x7fbbb409fd40 0x7fbbb413cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:06.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.686+0000 7fbbbbdd0640 1 --2- 172.21.15.100:0/2147831131 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbb40a42c0 0x7fbbb413f100 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:06.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.686+0000 7fbbbbdd0640 1 -- 172.21.15.100:0/2147831131 >> 172.21.15.100:0/2147831131 conn(0x7fbbb40a83f0 msgr2=0x7fbbb40a48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:06.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.686+0000 7fbbbbdd0640 1 -- 172.21.15.100:0/2147831131 shutdown_connections 2024-09-13T11:53:06.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.686+0000 7fbbbbdd0640 1 -- 172.21.15.100:0/2147831131 wait complete. 2024-09-13T11:53:06.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.687+0000 7fbbbbdd0640 10 monclient: build_initial_monmap 2024-09-13T11:53:06.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.687+0000 7fbbbbdd0640 10 monclient: monmap: 2024-09-13T11:53:06.707 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:06.707 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:06.707 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:06.707 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:06.708 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:06.708 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:06.708 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:06.708 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:06.708 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:06.708 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:06.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.687+0000 7fbbbbdd0640 1 Processor -- start 2024-09-13T11:53:06.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.687+0000 7fbbbbdd0640 1 -- start start 2024-09-13T11:53:06.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.687+0000 7fbbbbdd0640 10 monclient: init 2024-09-13T11:53:06.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.687+0000 7fbbbbdd0640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:06.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.687+0000 7fbbbbdd0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:53:06.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.687+0000 7fbbbbdd0640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbb40a37d0 0x7fbbb413a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:06.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.687+0000 7fbbbbdd0640 10 monclient(hunting): picked mon.noname-c con 0x7fbbb40a37d0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:06.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.687+0000 7fbbbbdd0640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbbb40a42c0 0x7fbbb413ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:06.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.687+0000 7fbbbbdd0640 10 monclient(hunting): picked mon.noname-a con 0x7fbbb40a42c0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:06.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbbbdd0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbbb413b1c0 0x7fbbb4136f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:06.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbbbdd0640 10 monclient(hunting): picked mon.noname-b con 0x7fbbb413b1c0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:06.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbbbdd0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:06.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbbbdd0640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb40a5240 con 0x7fbbb40a42c0 2024-09-13T11:53:06.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbbbdd0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:06.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbbbdd0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb4070310 con 0x7fbbb413b1c0 2024-09-13T11:53:06.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbbbdd0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:06.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbbbdd0640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fbbb40a53c0 con 0x7fbbb40a37d0 2024-09-13T11:53:06.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbbbdd0640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:06.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbbbdd0640 10 monclient(hunting): authenticate will time out at 767.281036s 2024-09-13T11:53:06.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb9b6c640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbb40a37d0 0x7fbbb413a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:06.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb936b640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbbb40a42c0 0x7fbbb413ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:06.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb9b6c640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbb40a37d0 0x7fbbb413a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47224/0 (socket says 172.21.15.100:47224) 2024-09-13T11:53:06.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb9b6c640 1 -- 172.21.15.100:0/962328386 learned_addr learned my addr 172.21.15.100:0/962328386 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:06.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb9b6c640 10 monclient(hunting): get_auth_request con 0x7fbbb40a37d0 auth_method 0 2024-09-13T11:53:06.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb9b6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:06.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb9b6c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:06.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb9b6c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:06.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb936b640 10 monclient(hunting): get_auth_request con 0x7fbbb40a42c0 auth_method 0 2024-09-13T11:53:06.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb936b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:06.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb936b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:06.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb936b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:06.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb9b6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:06.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb9b6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:06.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb9b6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:06.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb936b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:06.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb936b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:06.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb936b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:06.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb9b6c640 10 monclient(hunting): handle_auth_done global_id 4214 payload 824 2024-09-13T11:53:06.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbba36d640 1 --2- 172.21.15.100:0/962328386 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbbb413b1c0 0x7fbbb4136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:06.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb9b6c640 1 -- 172.21.15.100:0/962328386 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbbb413b1c0 msgr2=0x7fbbb4136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:06.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb9b6c640 1 --2- 172.21.15.100:0/962328386 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbbb413b1c0 0x7fbbb4136f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:06.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb9b6c640 1 -- 172.21.15.100:0/962328386 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbbb40a42c0 msgr2=0x7fbbb413ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:06.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb9b6c640 1 --2- 172.21.15.100:0/962328386 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbbb40a42c0 0x7fbbb413ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:06.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb9b6c640 10 monclient: _finish_hunting 0 2024-09-13T11:53:06.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb9b6c640 1 monclient: found mon.noname-c 2024-09-13T11:53:06.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb9b6c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:06.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb9b6c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:53:06.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb9b6c640 1 -- 172.21.15.100:0/962328386 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb413e280 con 0x7fbbb40a37d0 2024-09-13T11:53:06.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb936b640 1 --2- 172.21.15.100:0/962328386 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbbb40a42c0 0x7fbbb413ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:06.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbb9b6c640 1 --2- 172.21.15.100:0/962328386 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbb40a37d0 0x7fbbb413a740 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbba800c500 tx=0x7fbba800c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:06.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbba67fc640 1 -- 172.21.15.100:0/962328386 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba8017070 con 0x7fbbb40a37d0 2024-09-13T11:53:06.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbba67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:06.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbba67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:06.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbba67fc640 10 monclient: dump: 2024-09-13T11:53:06.712 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:06.712 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:06.712 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:06.712 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:06.712 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:06.712 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:06.712 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:06.712 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:06.712 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:06.713 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:06.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbba67fc640 10 monclient: _finish_auth 0 2024-09-13T11:53:06.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbba67fc640 10 monclient: _check_auth_tickets 2024-09-13T11:53:06.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbba67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:06.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbbbdd0640 5 monclient: authenticate success, global_id 4214 2024-09-13T11:53:06.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbba67fc640 1 -- 172.21.15.100:0/962328386 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbba8005ce0 con 0x7fbbb40a37d0 2024-09-13T11:53:06.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbba67fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:06.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbba67fc640 1 -- 172.21.15.100:0/962328386 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba8007270 con 0x7fbbb40a37d0 2024-09-13T11:53:06.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbbbdd0640 10 monclient: _renew_subs 2024-09-13T11:53:06.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbbbdd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:06.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbbbdd0640 1 -- 172.21.15.100:0/962328386 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbb413ea10 con 0x7fbbb40a37d0 2024-09-13T11:53:06.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbba67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:06.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbba67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:06.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbba7fff640 4 set_mon_vals no callback set 2024-09-13T11:53:06.714 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbba67fc640 10 monclient: dump: 2024-09-13T11:53:06.714 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:06.714 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:06.714 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:06.714 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:06.714 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:06.714 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:06.714 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:06.714 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:06.714 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:06.714 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:06.714 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbbbdd0640 10 monclient: _renew_subs 2024-09-13T11:53:06.714 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbbbdd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:06.714 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.688+0000 7fbbbbdd0640 1 -- 172.21.15.100:0/962328386 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbb40a08c0 con 0x7fbbb40a37d0 2024-09-13T11:53:06.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.689+0000 7fbba67fc640 1 -- 172.21.15.100:0/962328386 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fbba801a070 con 0x7fbbb40a37d0 2024-09-13T11:53:06.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.689+0000 7fbbbbdd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:06.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.689+0000 7fbbbbdd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:06.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.689+0000 7fbbbbdd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:06.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.689+0000 7fbbbbdd0640 1 -- 172.21.15.100:0/962328386 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbb409f0b0 con 0x7fbbb40a37d0 2024-09-13T11:53:06.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.689+0000 7fbba67fc640 1 --2- 172.21.15.100:0/962328386 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fbb9803dfa0 0x7fbb98040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:06.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.689+0000 7fbba67fc640 1 -- 172.21.15.100:0/962328386 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7fbba8051910 con 0x7fbbb40a37d0 2024-09-13T11:53:06.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.689+0000 7fbbb936b640 1 --2- 172.21.15.100:0/962328386 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fbb9803dfa0 0x7fbb98040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:06.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.689+0000 7fbbb936b640 10 monclient: get_auth_request con 0x7fbb9803dfa0 auth_method 0 2024-09-13T11:53:06.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.690+0000 7fbbb936b640 1 --2- 172.21.15.100:0/962328386 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fbb9803dfa0 0x7fbb98040460 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4074800 tx=0x7fbbb0008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:06.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.694+0000 7fbba67fc640 1 -- 172.21.15.100:0/962328386 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fbba801b3b0 con 0x7fbbb40a37d0 2024-09-13T11:53:06.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.694+0000 7fbba67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:06.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.694+0000 7fbba67fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:06.914 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.913+0000 7fbbbbdd0640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-13T11:53:06.914 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.913+0000 7fbbbbdd0640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-13T11:53:06.915 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.913+0000 7fbbbbdd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:06.915 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.913+0000 7fbbbbdd0640 1 -- 172.21.15.100:0/962328386 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fbbb40a3bb0 con 0x7fbbb40a37d0 2024-09-13T11:53:06.916 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.914+0000 7fbba67fc640 1 -- 172.21.15.100:0/962328386 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+97043 (secure 0 0 0) 0x7fbba801b730 con 0x7fbbb40a37d0 2024-09-13T11:53:06.916 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.914+0000 7fbba67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-13T11:53:06.916 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.914+0000 7fbba67fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:06.918 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:53:06.920 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.917+0000 7fbbbbdd0640 1 -- 172.21.15.100:0/962328386 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fbb9803dfa0 msgr2=0x7fbb98040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:06.920 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.917+0000 7fbbbbdd0640 1 --2- 172.21.15.100:0/962328386 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fbb9803dfa0 0x7fbb98040460 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4074800 tx=0x7fbbb0008040 comp rx=0 tx=0).stop 2024-09-13T11:53:06.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.917+0000 7fbbbbdd0640 10 monclient: shutdown 2024-09-13T11:53:06.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.917+0000 7fbbbbdd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:06.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.917+0000 7fbbbbdd0640 1 -- 172.21.15.100:0/962328386 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbb40a37d0 msgr2=0x7fbbb413a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:06.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.917+0000 7fbbbbdd0640 1 --2- 172.21.15.100:0/962328386 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbb40a37d0 0x7fbbb413a740 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbba800c500 tx=0x7fbba800c9c0 comp rx=0 tx=0).stop 2024-09-13T11:53:06.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.917+0000 7fbbbbdd0640 1 -- 172.21.15.100:0/962328386 shutdown_connections 2024-09-13T11:53:06.922 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.917+0000 7fbbbbdd0640 1 --2- 172.21.15.100:0/962328386 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fbb9803dfa0 0x7fbb98040460 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:06.922 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.917+0000 7fbbbbdd0640 1 --2- 172.21.15.100:0/962328386 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbbb40a42c0 0x7fbbb413ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:06.922 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.917+0000 7fbbbbdd0640 1 --2- 172.21.15.100:0/962328386 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbbb413b1c0 0x7fbbb4136f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:06.922 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.917+0000 7fbbbbdd0640 1 --2- 172.21.15.100:0/962328386 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbb40a37d0 0x7fbbb413a740 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:06.922 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.917+0000 7fbbbbdd0640 1 -- 172.21.15.100:0/962328386 >> 172.21.15.100:0/962328386 conn(0x7fbbb40a83f0 msgr2=0x7fbbb409e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:06.922 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.917+0000 7fbbbbdd0640 1 -- 172.21.15.100:0/962328386 shutdown_connections 2024-09-13T11:53:06.923 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:06.917+0000 7fbbbbdd0640 1 -- 172.21.15.100:0/962328386 wait complete. 2024-09-13T11:53:06.929 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":5,"flags":0,"active_gid":4114,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":441789830},{"type":"v1","addr":"172.21.15.122:6825","nonce":441789830}]},"active_addr":"172.21.15.122:6825/441789830","active_change":"2024-09-13T11:52:51.202638+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":1204056135}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":1764498442}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":1565412657}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":3327779439}]}]} 2024-09-13T11:53:06.933 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-13T11:53:06.933 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-13T11:53:06.933 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T11:53:07.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.028+0000 7fa73d9f3640 10 monclient: get_monmap_and_config 2024-09-13T11:53:07.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.028+0000 7fa73d9f3640 10 monclient: build_initial_monmap 2024-09-13T11:53:07.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.028+0000 7fa73d9f3640 10 monclient: monmap: 2024-09-13T11:53:07.030 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:07.030 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.030 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:07.029718+0000 2024-09-13T11:53:07.030 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:07.029718+0000 2024-09-13T11:53:07.030 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:07.031 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.031 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:07.031 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:07.031 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:07.031 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:07.031 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:07.031 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.039 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.029+0000 7fa73d9f3640 1 Processor -- start 2024-09-13T11:53:07.039 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.029+0000 7fa73d9f3640 1 -- start start 2024-09-13T11:53:07.039 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.029+0000 7fa73d9f3640 10 monclient: init 2024-09-13T11:53:07.039 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa73d9f3640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:07.039 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa73d9f3640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-13T11:53:07.039 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa73d9f3640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa738137a60 0x7fa7380a2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.039 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa73d9f3640 10 monclient(hunting): picked mon.noname-a con 0x7fa738137a60 addr v2:172.21.15.100:3300/0 2024-09-13T11:53:07.039 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa73d9f3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa738137e40 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:07.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa73d9f3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa738138250 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:07.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa73d9f3640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7380a5770 con 0x7fa738138250 2024-09-13T11:53:07.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa73d9f3640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7380a5be0 con 0x7fa738137e40 2024-09-13T11:53:07.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa73d9f3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa73d9f3640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7fa7380a4410 con 0x7fa738137a60 2024-09-13T11:53:07.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa73d9f3640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:07.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa73d9f3640 10 monclient(hunting): authenticate will time out at 767.623176s 2024-09-13T11:53:07.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa736ffd640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa738137a60 0x7fa7380a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa7377fe640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fa738138250 0x7fa7380a5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:57876/0 (socket says 172.21.15.100:57876) 2024-09-13T11:53:07.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa7377fe640 1 -- 172.21.15.100:0/3287588843 learned_addr learned my addr 172.21.15.100:0/3287588843 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:53:07.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa736ffd640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa738137a60 0x7fa7380a2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49426/0 (socket says 172.21.15.100:49426) 2024-09-13T11:53:07.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa736ffd640 10 monclient(hunting): get_auth_request con 0x7fa738137a60 auth_method 0 2024-09-13T11:53:07.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa736ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa736ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa736ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa736ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa736ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa736ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa736ffd640 10 monclient(hunting): handle_auth_done global_id 4248 payload 274 2024-09-13T11:53:07.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa736ffd640 1 -- 172.21.15.100:0/3287588843 >> v1:172.21.15.122:6789/0 conn(0x7fa738137e40 legacy=0x7fa7380a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:07.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa736ffd640 1 -- 172.21.15.100:0/3287588843 >> v1:172.21.15.100:6789/0 conn(0x7fa738138250 legacy=0x7fa7380a5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa736ffd640 10 monclient: _finish_hunting 0 2024-09-13T11:53:07.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa736ffd640 1 monclient: found mon.noname-a 2024-09-13T11:53:07.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa736ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:07.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa736ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:07.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa736ffd640 1 -- 172.21.15.100:0/3287588843 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7380a76b0 con 0x7fa738137a60 2024-09-13T11:53:07.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa735ffb640 1 -- 172.21.15.100:0/3287588843 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1768463494 0 0) 0x7fa7380a5770 con 0x7fa738138250 2024-09-13T11:53:07.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.030+0000 7fa735ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:53:07.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa736ffd640 1 --2- 172.21.15.100:0/3287588843 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa738137a60 0x7fa7380a2980 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa7380a5db0 tx=0x7fa72802a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=690bdf4bf0b423f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa735ffb640 1 -- 172.21.15.100:0/3287588843 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa728037070 con 0x7fa738137a60 2024-09-13T11:53:07.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa735ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa735ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:07.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa735ffb640 10 monclient: dump: 2024-09-13T11:53:07.042 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.042 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.042 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.042 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.042 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.042 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.043 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.043 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.043 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.043 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa735ffb640 10 monclient: _finish_auth 0 2024-09-13T11:53:07.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa735ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:53:07.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa735ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:07.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa735ffb640 1 -- 172.21.15.100:0/3287588843 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa72802adb0 con 0x7fa738137a60 2024-09-13T11:53:07.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa73d9f3640 5 monclient: authenticate success, global_id 4248 2024-09-13T11:53:07.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa73d9f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:07.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa735ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:07.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa73d9f3640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:07.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa735ffb640 1 -- 172.21.15.100:0/3287588843 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa728032660 con 0x7fa738137a60 2024-09-13T11:53:07.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa735ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.044 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa73d9f3640 4 set_mon_vals no callback set 2024-09-13T11:53:07.044 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa735ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:07.044 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa735ffb640 10 monclient: dump: 2024-09-13T11:53:07.044 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.044 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.044 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.044 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.044 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.044 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.044 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.044 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.044 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.044 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa73d9f3640 10 monclient: shutdown 2024-09-13T11:53:07.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa73d9f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:07.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa73d9f3640 1 -- 172.21.15.100:0/3287588843 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa738137a60 msgr2=0x7fa7380a2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa73d9f3640 1 --2- 172.21.15.100:0/3287588843 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa738137a60 0x7fa7380a2980 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa7380a5db0 tx=0x7fa72802a060 comp rx=0 tx=0).stop 2024-09-13T11:53:07.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa73d9f3640 1 -- 172.21.15.100:0/3287588843 shutdown_connections 2024-09-13T11:53:07.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa73d9f3640 1 --2- 172.21.15.100:0/3287588843 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa738137a60 0x7fa7380a2980 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa73d9f3640 1 -- 172.21.15.100:0/3287588843 >> 172.21.15.100:0/3287588843 conn(0x7fa7380aa580 msgr2=0x7fa7380a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:07.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa73d9f3640 1 -- 172.21.15.100:0/3287588843 shutdown_connections 2024-09-13T11:53:07.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.031+0000 7fa73d9f3640 1 -- 172.21.15.100:0/3287588843 wait complete. 2024-09-13T11:53:07.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.032+0000 7fa73d9f3640 10 monclient: build_initial_monmap 2024-09-13T11:53:07.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.032+0000 7fa73d9f3640 10 monclient: monmap: 2024-09-13T11:53:07.045 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:07.045 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:07.046 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:07.046 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:07.046 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:07.046 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.046 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:07.046 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:07.046 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:07.046 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.032+0000 7fa73d9f3640 1 Processor -- start 2024-09-13T11:53:07.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.032+0000 7fa73d9f3640 1 -- start start 2024-09-13T11:53:07.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.032+0000 7fa73d9f3640 10 monclient: init 2024-09-13T11:53:07.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa73d9f3640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:07.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa73d9f3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:53:07.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa73d9f3640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa738137a60 0x7fa73813c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa73d9f3640 10 monclient(hunting): picked mon.noname-a con 0x7fa738137a60 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:07.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa73d9f3640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa738137e40 0x7fa73813c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa73d9f3640 10 monclient(hunting): picked mon.noname-c con 0x7fa738137e40 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:07.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa73d9f3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa738138250 0x7fa73813cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa73d9f3640 10 monclient(hunting): picked mon.noname-b con 0x7fa738138250 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:07.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa73d9f3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa73d9f3640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fa73806fd80 con 0x7fa738137a60 2024-09-13T11:53:07.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa73d9f3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa73d9f3640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa7380a1700 con 0x7fa738138250 2024-09-13T11:53:07.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa73d9f3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa73d9f3640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fa73806ff00 con 0x7fa738137e40 2024-09-13T11:53:07.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa73d9f3640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:07.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa73d9f3640 10 monclient(hunting): authenticate will time out at 767.626340s 2024-09-13T11:53:07.048 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa736ffd640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa738137a60 0x7fa73813c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.048 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa736ffd640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa738137a60 0x7fa73813c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49442/0 (socket says 172.21.15.100:49442) 2024-09-13T11:53:07.048 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa736ffd640 1 -- 172.21.15.100:0/1868582119 learned_addr learned my addr 172.21.15.100:0/1868582119 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:07.048 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa736ffd640 10 monclient(hunting): get_auth_request con 0x7fa738137a60 auth_method 0 2024-09-13T11:53:07.048 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa736ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.048 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa736ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.048 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa736ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.048 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa7367fc640 1 --2- 172.21.15.100:0/1868582119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa738137e40 0x7fa73813c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.048 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa7367fc640 10 monclient(hunting): get_auth_request con 0x7fa738137e40 auth_method 0 2024-09-13T11:53:07.048 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa7367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.048 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa7367fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.048 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa7367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.048 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa736ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa736ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa736ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa7367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa7367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa7367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa7377fe640 1 --2- 172.21.15.100:0/1868582119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa738138250 0x7fa73813cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa7377fe640 10 monclient(hunting): get_auth_request con 0x7fa738138250 auth_method 0 2024-09-13T11:53:07.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa7377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa7377fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa7377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa736ffd640 10 monclient(hunting): handle_auth_done global_id 4254 payload 824 2024-09-13T11:53:07.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa736ffd640 1 -- 172.21.15.100:0/1868582119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa738137e40 msgr2=0x7fa73813c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa736ffd640 1 --2- 172.21.15.100:0/1868582119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa738137e40 0x7fa73813c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa736ffd640 1 -- 172.21.15.100:0/1868582119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa738138250 msgr2=0x7fa73813cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa736ffd640 1 --2- 172.21.15.100:0/1868582119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa738138250 0x7fa73813cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa736ffd640 10 monclient: _finish_hunting 0 2024-09-13T11:53:07.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa736ffd640 1 monclient: found mon.noname-a 2024-09-13T11:53:07.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa736ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:07.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa736ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:07.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa736ffd640 1 -- 172.21.15.100:0/1868582119 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7380a1140 con 0x7fa738137a60 2024-09-13T11:53:07.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa7367fc640 1 --2- 172.21.15.100:0/1868582119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa738137e40 0x7fa73813c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:07.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa7377fe640 1 --2- 172.21.15.100:0/1868582119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa738138250 0x7fa73813cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:53:07.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.033+0000 7fa736ffd640 1 --2- 172.21.15.100:0/1868582119 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa738137a60 0x7fa73813c300 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa728009890 tx=0x7fa7280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.034+0000 7fa71ffff640 1 -- 172.21.15.100:0/1868582119 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa728042070 con 0x7fa738137a60 2024-09-13T11:53:07.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.034+0000 7fa71ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.034+0000 7fa71ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:07.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.034+0000 7fa71ffff640 10 monclient: dump: 2024-09-13T11:53:07.051 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.051 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.051 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.051 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.051 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.051 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.051 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.051 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.051 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.051 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.034+0000 7fa71ffff640 10 monclient: _finish_auth 0 2024-09-13T11:53:07.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.034+0000 7fa71ffff640 10 monclient: _check_auth_tickets 2024-09-13T11:53:07.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.034+0000 7fa71ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:07.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.034+0000 7fa73d9f3640 5 monclient: authenticate success, global_id 4254 2024-09-13T11:53:07.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.034+0000 7fa71ffff640 1 -- 172.21.15.100:0/1868582119 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa72803e670 con 0x7fa738137a60 2024-09-13T11:53:07.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.034+0000 7fa71ffff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:07.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.034+0000 7fa71ffff640 1 -- 172.21.15.100:0/1868582119 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa728037040 con 0x7fa738137a60 2024-09-13T11:53:07.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.034+0000 7fa71ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.034+0000 7fa71ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:07.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.034+0000 7fa71ffff640 10 monclient: dump: 2024-09-13T11:53:07.052 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.052 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.052 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.052 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.053 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.053 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.053 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.053 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.053 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.053 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.034+0000 7fa73d9f3640 10 monclient: _renew_subs 2024-09-13T11:53:07.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.034+0000 7fa73d9f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:07.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.034+0000 7fa73d9f3640 1 -- 172.21.15.100:0/1868582119 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa73813fb60 con 0x7fa738137a60 2024-09-13T11:53:07.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.034+0000 7fa7357fa640 4 set_mon_vals no callback set 2024-09-13T11:53:07.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.034+0000 7fa73d9f3640 10 monclient: _renew_subs 2024-09-13T11:53:07.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.034+0000 7fa73d9f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:07.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.034+0000 7fa73d9f3640 1 -- 172.21.15.100:0/1868582119 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa73813fe40 con 0x7fa738137a60 2024-09-13T11:53:07.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.035+0000 7fa71ffff640 1 -- 172.21.15.100:0/1868582119 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fa728045070 con 0x7fa738137a60 2024-09-13T11:53:07.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.035+0000 7fa73d9f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:07.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.035+0000 7fa73d9f3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:07.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.035+0000 7fa73d9f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:07.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.035+0000 7fa73d9f3640 1 -- 172.21.15.100:0/1868582119 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7380a2980 con 0x7fa738137a60 2024-09-13T11:53:07.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.035+0000 7fa71ffff640 1 --2- 172.21.15.100:0/1868582119 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fa72403dfa0 0x7fa724040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.035+0000 7fa71ffff640 1 -- 172.21.15.100:0/1868582119 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7fa728072e90 con 0x7fa738137a60 2024-09-13T11:53:07.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.035+0000 7fa7367fc640 1 --2- 172.21.15.100:0/1868582119 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fa72403dfa0 0x7fa724040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.035+0000 7fa7367fc640 10 monclient: get_auth_request con 0x7fa72403dfa0 auth_method 0 2024-09-13T11:53:07.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.036+0000 7fa7367fc640 1 --2- 172.21.15.100:0/1868582119 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fa72403dfa0 0x7fa724040460 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa73809fd60 tx=0x7fa72c007e30 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.055 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.039+0000 7fa71ffff640 1 -- 172.21.15.100:0/1868582119 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fa7380a1140 con 0x7fa738137a60 2024-09-13T11:53:07.055 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.039+0000 7fa71ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:07.055 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.039+0000 7fa71ffff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:07.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.213+0000 7fa73d9f3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:53:07.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.213+0000 7fa73d9f3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:53:07.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.213+0000 7fa73d9f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:07.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.213+0000 7fa73d9f3640 1 -- 172.21.15.100:0/1868582119 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa73813ddc0 con 0x7fa738137a60 2024-09-13T11:53:07.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.214+0000 7fa71ffff640 1 -- 172.21.15.100:0/1868582119 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12401 (secure 0 0 0) 0x7fa728045c60 con 0x7fa738137a60 2024-09-13T11:53:07.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.214+0000 7fa71ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:53:07.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.214+0000 7fa71ffff640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:07.216 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:53:07.216 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":18,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","created":"2024-09-13T11:52:46.158794+0000","modified":"2024-09-13T11:53:06.262248+0000","last_up_change":"2024-09-13T11:52:56.225375+0000","last_in_change":"2024-09-13T11:52:47.800207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T11:52:55.233767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T11:53:03.003491+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac784459-dbff-42ed-941e-70c741392dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":92840425},{"type":"v1","addr":"172.21.15.100:6817","nonce":92840425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":92840425},{"type":"v1","addr":"172.21.15.100:6819","nonce":92840425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6822","nonce":92840425},{"type":"v1","addr":"172.21.15.100:6823","nonce":92840425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":92840425},{"type":"v1","addr":"172.21.15.100:6821","nonce":92840425}]},"public_addr":"172.21.15.100:6817/92840425","cluster_addr":"172.21.15.100:6819/92840425","heartbeat_back_addr":"172.21.15.100:6823/92840425","heartbeat_front_addr":"172.21.15.100:6821/92840425","state":["exists","up"]},{"osd":1,"uuid":"8d8d39e7-e8f1-499a-a01b-e7ba49844eae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":1385319817},{"type":"v1","addr":"172.21.15.100:6809","nonce":1385319817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":1385319817},{"type":"v1","addr":"172.21.15.100:6811","nonce":1385319817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":1385319817},{"type":"v1","addr":"172.21.15.100:6815","nonce":1385319817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":1385319817},{"type":"v1","addr":"172.21.15.100:6813","nonce":1385319817}]},"public_addr":"172.21.15.100:6809/1385319817","cluster_addr":"172.21.15.100:6811/1385319817","heartbeat_back_addr":"172.21.15.100:6815/1385319817","heartbeat_front_addr":"172.21.15.100:6813/1385319817","state":["exists","up"]},{"osd":2,"uuid":"e0bd3e30-0ff3-4067-83d3-ef8d84b29097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":2354314235},{"type":"v1","addr":"172.21.15.100:6801","nonce":2354314235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":2354314235},{"type":"v1","addr":"172.21.15.100:6803","nonce":2354314235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":2354314235},{"type":"v1","addr":"172.21.15.100:6807","nonce":2354314235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":2354314235},{"type":"v1","addr":"172.21.15.100:6805","nonce":2354314235}]},"public_addr":"172.21.15.100:6801/2354314235","cluster_addr":"172.21.15.100:6803/2354314235","heartbeat_back_addr":"172.21.15.100:6807/2354314235","heartbeat_front_addr":"172.21.15.100:6805/2354314235","state":["exists","up"]},{"osd":3,"uuid":"eca4429b-38cd-488a-b31d-c5856d701bde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3206511700},{"type":"v1","addr":"172.21.15.122:6809","nonce":3206511700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3206511700},{"type":"v1","addr":"172.21.15.122:6811","nonce":3206511700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3206511700},{"type":"v1","addr":"172.21.15.122:6815","nonce":3206511700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3206511700},{"type":"v1","addr":"172.21.15.122:6813","nonce":3206511700}]},"public_addr":"172.21.15.122:6809/3206511700","cluster_addr":"172.21.15.122:6811/3206511700","heartbeat_back_addr":"172.21.15.122:6815/3206511700","heartbeat_front_addr":"172.21.15.122:6813/3206511700","state":["exists","up"]},{"osd":4,"uuid":"b7ddc8f1-38d5-496a-9bf7-181eab7e5516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":777240702},{"type":"v1","addr":"172.21.15.122:6817","nonce":777240702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":777240702},{"type":"v1","addr":"172.21.15.122:6819","nonce":777240702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":777240702},{"type":"v1","addr":"172.21.15.122:6823","nonce":777240702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":777240702},{"type":"v1","addr":"172.21.15.122:6821","nonce":777240702}]},"public_addr":"172.21.15.122:6817/777240702","cluster_addr":"172.21.15.122:6819/777240702","heartbeat_back_addr":"172.21.15.122:6823/777240702","heartbeat_front_addr":"172.21.15.122:6821/777240702","state":["exists","up"]},{"osd":5,"uuid":"9010f8e0-c97b-451c-8b29-3ccb99e982a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":904725860},{"type":"v1","addr":"172.21.15.122:6801","nonce":904725860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":904725860},{"type":"v1","addr":"172.21.15.122:6803","nonce":904725860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":904725860},{"type":"v1","addr":"172.21.15.122:6807","nonce":904725860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":904725860},{"type":"v1","addr":"172.21.15.122:6805","nonce":904725860}]},"public_addr":"172.21.15.122:6801/904725860","cluster_addr":"172.21.15.122:6803/904725860","heartbeat_back_addr":"172.21.15.122:6807/904725860","heartbeat_front_addr":"172.21.15.122:6805/904725860","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:54.752725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:53.791663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:53.535397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:54.348980+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:54.658725+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:53.932256+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:53:07.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.216+0000 7fa73d9f3640 1 -- 172.21.15.100:0/1868582119 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fa72403dfa0 msgr2=0x7fa724040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.216+0000 7fa73d9f3640 1 --2- 172.21.15.100:0/1868582119 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fa72403dfa0 0x7fa724040460 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa73809fd60 tx=0x7fa72c007e30 comp rx=0 tx=0).stop 2024-09-13T11:53:07.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.216+0000 7fa73d9f3640 10 monclient: shutdown 2024-09-13T11:53:07.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.216+0000 7fa73d9f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:07.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.216+0000 7fa73d9f3640 1 -- 172.21.15.100:0/1868582119 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa738137a60 msgr2=0x7fa73813c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.216+0000 7fa73d9f3640 1 --2- 172.21.15.100:0/1868582119 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa738137a60 0x7fa73813c300 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa728009890 tx=0x7fa7280046c0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.216+0000 7fa73d9f3640 1 -- 172.21.15.100:0/1868582119 shutdown_connections 2024-09-13T11:53:07.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.216+0000 7fa73d9f3640 1 --2- 172.21.15.100:0/1868582119 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fa72403dfa0 0x7fa724040460 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.216+0000 7fa73d9f3640 1 --2- 172.21.15.100:0/1868582119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa738138250 0x7fa73813cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.216+0000 7fa73d9f3640 1 --2- 172.21.15.100:0/1868582119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa738137e40 0x7fa73813c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.216+0000 7fa73d9f3640 1 --2- 172.21.15.100:0/1868582119 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa738137a60 0x7fa73813c300 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.216+0000 7fa73d9f3640 1 -- 172.21.15.100:0/1868582119 >> 172.21.15.100:0/1868582119 conn(0x7fa7380aa580 msgr2=0x7fa73809da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:07.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.216+0000 7fa73d9f3640 1 -- 172.21.15.100:0/1868582119 shutdown_connections 2024-09-13T11:53:07.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.216+0000 7fa73d9f3640 1 -- 172.21.15.100:0/1868582119 wait complete. 2024-09-13T11:53:07.228 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-13T11:53:07.229 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T11:53:07.326 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.325+0000 7f06334b8640 10 monclient: get_monmap_and_config 2024-09-13T11:53:07.326 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.325+0000 7f06334b8640 10 monclient: build_initial_monmap 2024-09-13T11:53:07.326 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.325+0000 7f06334b8640 10 monclient: monmap: 2024-09-13T11:53:07.327 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:07.327 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.327 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:07.326081+0000 2024-09-13T11:53:07.327 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:07.326081+0000 2024-09-13T11:53:07.327 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:07.327 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.327 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:07.327 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:07.327 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:07.327 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:07.327 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:07.327 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.334 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.325+0000 7f06334b8640 1 Processor -- start 2024-09-13T11:53:07.334 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f06334b8640 1 -- start start 2024-09-13T11:53:07.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f06334b8640 10 monclient: init 2024-09-13T11:53:07.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f06334b8640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:07.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f06334b8640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-13T11:53:07.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f06334b8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f062c137a60 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:07.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f06334b8640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f062c0a2980 0x7f062c13c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f06334b8640 10 monclient(hunting): picked mon.noname-b con 0x7f062c0a2980 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:07.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f06334b8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f062c13ce90 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:07.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f06334b8640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f062c138550 con 0x7f062c13ce90 2024-09-13T11:53:07.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f06334b8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f062c0a5be0 con 0x7f062c137a60 2024-09-13T11:53:07.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f06334b8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f06334b8640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f062c0a4410 con 0x7f062c0a2980 2024-09-13T11:53:07.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f06334b8640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:07.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f06334b8640 10 monclient(hunting): authenticate will time out at 767.919502s 2024-09-13T11:53:07.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f0630a53640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f062c0a2980 0x7f062c13c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f0631a55640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f062c13ce90 0x7f062c13f280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:57890/0 (socket says 172.21.15.100:57890) 2024-09-13T11:53:07.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f0631a55640 1 -- 172.21.15.100:0/656082659 learned_addr learned my addr 172.21.15.100:0/656082659 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:53:07.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f0630a53640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f062c0a2980 0x7f062c13c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47250/0 (socket says 172.21.15.100:47250) 2024-09-13T11:53:07.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f0630a53640 10 monclient(hunting): get_auth_request con 0x7f062c0a2980 auth_method 0 2024-09-13T11:53:07.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f0630a53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f0630a53640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f0630a53640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f0630a53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f0630a53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f0630a53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f0630a53640 10 monclient(hunting): handle_auth_done global_id 4220 payload 274 2024-09-13T11:53:07.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f061ffff640 1 -- 172.21.15.100:0/656082659 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3729987148 0 0) 0x7f062c138550 con 0x7f062c13ce90 2024-09-13T11:53:07.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f0630a53640 1 -- 172.21.15.100:0/656082659 >> v1:172.21.15.122:6789/0 conn(0x7f062c137a60 legacy=0x7f062c137e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:07.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f0630a53640 1 -- 172.21.15.100:0/656082659 >> v1:172.21.15.100:6789/0 conn(0x7f062c13ce90 legacy=0x7f062c13f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f0630a53640 10 monclient: _finish_hunting 0 2024-09-13T11:53:07.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f0630a53640 1 monclient: found mon.noname-b 2024-09-13T11:53:07.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f0630a53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:07.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f0630a53640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:53:07.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f0630a53640 1 -- 172.21.15.100:0/656082659 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f062c0a76b0 con 0x7f062c0a2980 2024-09-13T11:53:07.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.326+0000 7f061ffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:53:07.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.327+0000 7f0630a53640 1 --2- 172.21.15.100:0/656082659 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f062c0a2980 0x7f062c13c950 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f06280097d0 tx=0x7f062802a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2fb0b7753728de3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.327+0000 7f061ffff640 1 -- 172.21.15.100:0/656082659 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0628037070 con 0x7f062c0a2980 2024-09-13T11:53:07.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.327+0000 7f061ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.327+0000 7f061ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:07.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.327+0000 7f061ffff640 10 monclient: dump: 2024-09-13T11:53:07.338 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.338 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.338 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.338 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.338 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.338 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.338 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.338 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.338 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.338 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.338 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.327+0000 7f061ffff640 10 monclient: _finish_auth 0 2024-09-13T11:53:07.338 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.327+0000 7f061ffff640 10 monclient: _check_auth_tickets 2024-09-13T11:53:07.338 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.327+0000 7f061ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:07.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.327+0000 7f061ffff640 1 -- 172.21.15.100:0/656082659 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f062802ad90 con 0x7f062c0a2980 2024-09-13T11:53:07.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.327+0000 7f06334b8640 5 monclient: authenticate success, global_id 4220 2024-09-13T11:53:07.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.327+0000 7f06334b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:07.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.327+0000 7f061ffff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:07.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.327+0000 7f06334b8640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:07.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.327+0000 7f061ffff640 1 -- 172.21.15.100:0/656082659 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0628032660 con 0x7f062c0a2980 2024-09-13T11:53:07.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.327+0000 7f06334b8640 4 set_mon_vals no callback set 2024-09-13T11:53:07.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.327+0000 7f061ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.327+0000 7f061ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:07.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.327+0000 7f061ffff640 10 monclient: dump: 2024-09-13T11:53:07.339 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.339 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.339 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.339 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.340 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.340 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.340 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.340 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.340 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.340 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.328+0000 7f06334b8640 10 monclient: shutdown 2024-09-13T11:53:07.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.328+0000 7f06334b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:07.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.328+0000 7f06334b8640 1 -- 172.21.15.100:0/656082659 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f062c0a2980 msgr2=0x7f062c13c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.328+0000 7f06334b8640 1 --2- 172.21.15.100:0/656082659 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f062c0a2980 0x7f062c13c950 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f06280097d0 tx=0x7f062802a040 comp rx=0 tx=0).stop 2024-09-13T11:53:07.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.328+0000 7f06334b8640 1 -- 172.21.15.100:0/656082659 shutdown_connections 2024-09-13T11:53:07.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.328+0000 7f06334b8640 1 --2- 172.21.15.100:0/656082659 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f062c0a2980 0x7f062c13c950 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.328+0000 7f06334b8640 1 -- 172.21.15.100:0/656082659 >> 172.21.15.100:0/656082659 conn(0x7f062c0aa580 msgr2=0x7f062c0a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:07.341 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.328+0000 7f06334b8640 1 -- 172.21.15.100:0/656082659 shutdown_connections 2024-09-13T11:53:07.341 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.328+0000 7f06334b8640 1 -- 172.21.15.100:0/656082659 wait complete. 2024-09-13T11:53:07.341 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.328+0000 7f06334b8640 10 monclient: build_initial_monmap 2024-09-13T11:53:07.341 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.328+0000 7f06334b8640 10 monclient: monmap: 2024-09-13T11:53:07.341 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:07.341 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:07.341 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:07.341 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:07.341 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:07.341 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.341 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:07.341 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:07.341 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:07.342 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f06334b8640 1 Processor -- start 2024-09-13T11:53:07.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f06334b8640 1 -- start start 2024-09-13T11:53:07.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f06334b8640 10 monclient: init 2024-09-13T11:53:07.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f06334b8640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:07.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f06334b8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:53:07.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f06334b8640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f062c0a2980 0x7f062c09eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f06334b8640 10 monclient(hunting): picked mon.noname-a con 0x7f062c0a2980 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:07.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f06334b8640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f062c137a60 0x7f062c136700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f06334b8640 10 monclient(hunting): picked mon.noname-c con 0x7f062c137a60 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:07.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f06334b8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f062c13ce90 0x7f062c09c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f06334b8640 10 monclient(hunting): picked mon.noname-b con 0x7f062c13ce90 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:07.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f06334b8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f06334b8640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f062c06fd80 con 0x7f062c0a2980 2024-09-13T11:53:07.343 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f06334b8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.343 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f06334b8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f062c0a1700 con 0x7f062c13ce90 2024-09-13T11:53:07.343 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f06334b8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.343 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f06334b8640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f062c06ff00 con 0x7f062c137a60 2024-09-13T11:53:07.343 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f06334b8640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:07.343 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f06334b8640 10 monclient(hunting): authenticate will time out at 767.922692s 2024-09-13T11:53:07.343 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f0631254640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f062c0a2980 0x7f062c09eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.343 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f0630a53640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f062c137a60 0x7f062c136700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.343 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f0631254640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f062c0a2980 0x7f062c09eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49446/0 (socket says 172.21.15.100:49446) 2024-09-13T11:53:07.343 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f0630a53640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f062c137a60 0x7f062c136700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47252/0 (socket says 172.21.15.100:47252) 2024-09-13T11:53:07.343 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f0631254640 1 -- 172.21.15.100:0/3020730363 learned_addr learned my addr 172.21.15.100:0/3020730363 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:07.343 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f0631254640 10 monclient(hunting): get_auth_request con 0x7f062c0a2980 auth_method 0 2024-09-13T11:53:07.343 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f0631254640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f0631254640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f0631254640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f0630a53640 10 monclient(hunting): get_auth_request con 0x7f062c137a60 auth_method 0 2024-09-13T11:53:07.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f0630a53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f0630a53640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f0630a53640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f0631254640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f0631254640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f0631254640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f0630a53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f0630a53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f0630a53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f0631a55640 1 --2- 172.21.15.100:0/3020730363 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f062c13ce90 0x7f062c09c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f0631a55640 10 monclient(hunting): get_auth_request con 0x7f062c13ce90 auth_method 0 2024-09-13T11:53:07.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f0631a55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f0631a55640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.329+0000 7f0631a55640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f0630a53640 10 monclient(hunting): handle_auth_done global_id 4223 payload 824 2024-09-13T11:53:07.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f0630a53640 1 -- 172.21.15.100:0/3020730363 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f062c13ce90 msgr2=0x7f062c09c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f0630a53640 1 --2- 172.21.15.100:0/3020730363 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f062c13ce90 0x7f062c09c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f0630a53640 1 -- 172.21.15.100:0/3020730363 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f062c0a2980 msgr2=0x7f062c09eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f0630a53640 1 --2- 172.21.15.100:0/3020730363 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f062c0a2980 0x7f062c09eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f0630a53640 10 monclient: _finish_hunting 0 2024-09-13T11:53:07.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f0630a53640 1 monclient: found mon.noname-c 2024-09-13T11:53:07.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f0630a53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:07.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f0630a53640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:53:07.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f0630a53640 1 -- 172.21.15.100:0/3020730363 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f062c09f830 con 0x7f062c137a60 2024-09-13T11:53:07.346 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f0631254640 1 --2- 172.21.15.100:0/3020730363 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f062c0a2980 0x7f062c09eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:07.346 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f0631a55640 1 --2- 172.21.15.100:0/3020730363 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f062c13ce90 0x7f062c09c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:53:07.346 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f0630a53640 1 --2- 172.21.15.100:0/3020730363 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f062c137a60 0x7f062c136700 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0628031000 tx=0x7f06280046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.346 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f061dffb640 1 -- 172.21.15.100:0/3020730363 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0628041070 con 0x7f062c137a60 2024-09-13T11:53:07.346 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f061dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.346 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f061dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:07.346 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f061dffb640 10 monclient: dump: 2024-09-13T11:53:07.346 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.346 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.346 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.346 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.346 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.346 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.347 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.347 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.347 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.347 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.347 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f061dffb640 10 monclient: _finish_auth 0 2024-09-13T11:53:07.347 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f061dffb640 10 monclient: _check_auth_tickets 2024-09-13T11:53:07.347 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f061dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:07.347 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f061dffb640 1 -- 172.21.15.100:0/3020730363 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f062803c670 con 0x7f062c137a60 2024-09-13T11:53:07.347 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f06334b8640 5 monclient: authenticate success, global_id 4223 2024-09-13T11:53:07.347 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f06334b8640 10 monclient: _renew_subs 2024-09-13T11:53:07.347 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f06334b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:07.347 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f06334b8640 1 -- 172.21.15.100:0/3020730363 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f062c0a06a0 con 0x7f062c137a60 2024-09-13T11:53:07.347 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f061dffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:07.347 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f061dffb640 1 -- 172.21.15.100:0/3020730363 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0628037040 con 0x7f062c137a60 2024-09-13T11:53:07.348 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f061dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.348 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f061dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:07.348 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f061dffb640 10 monclient: dump: 2024-09-13T11:53:07.348 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.348 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.348 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.348 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.348 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.348 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.348 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.348 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.348 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.348 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f06334b8640 10 monclient: _renew_subs 2024-09-13T11:53:07.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f061f7fe640 4 set_mon_vals no callback set 2024-09-13T11:53:07.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f06334b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:07.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.330+0000 7f06334b8640 1 -- 172.21.15.100:0/3020730363 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f062c0a03c0 con 0x7f062c137a60 2024-09-13T11:53:07.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.331+0000 7f061dffb640 1 -- 172.21.15.100:0/3020730363 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f0628044070 con 0x7f062c137a60 2024-09-13T11:53:07.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.331+0000 7f06334b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:07.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.331+0000 7f06334b8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:07.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.331+0000 7f06334b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:07.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.331+0000 7f06334b8640 1 -- 172.21.15.100:0/3020730363 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f062c0a86c0 con 0x7f062c137a60 2024-09-13T11:53:07.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.331+0000 7f061dffb640 1 --2- 172.21.15.100:0/3020730363 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f061003dfd0 0x7f0610040490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.332+0000 7f061dffb640 1 -- 172.21.15.100:0/3020730363 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7f0628072cf0 con 0x7f062c137a60 2024-09-13T11:53:07.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.332+0000 7f0631254640 1 --2- 172.21.15.100:0/3020730363 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f061003dfd0 0x7f0610040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.332+0000 7f0631254640 10 monclient: get_auth_request con 0x7f061003dfd0 auth_method 0 2024-09-13T11:53:07.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.332+0000 7f0631254640 1 --2- 172.21.15.100:0/3020730363 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f061003dfd0 0x7f0610040490 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f06200043e0 tx=0x7f0620007ef0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.350 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.336+0000 7f061dffb640 1 -- 172.21.15.100:0/3020730363 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f062c0a86c0 con 0x7f062c137a60 2024-09-13T11:53:07.350 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.336+0000 7f061dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:07.350 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.336+0000 7f061dffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:07.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.510+0000 7f06334b8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:53:07.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.510+0000 7f06334b8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:53:07.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.510+0000 7f06334b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:07.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.510+0000 7f06334b8640 1 -- 172.21.15.100:0/3020730363 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f062c137e40 con 0x7f062c137a60 2024-09-13T11:53:07.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.511+0000 7f061dffb640 1 -- 172.21.15.100:0/3020730363 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12401 (secure 0 0 0) 0x7f0628044d70 con 0x7f062c137a60 2024-09-13T11:53:07.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.511+0000 7f061dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:53:07.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.511+0000 7f061dffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:07.513 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:53:07.513 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":18,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","created":"2024-09-13T11:52:46.158794+0000","modified":"2024-09-13T11:53:06.262248+0000","last_up_change":"2024-09-13T11:52:56.225375+0000","last_in_change":"2024-09-13T11:52:47.800207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T11:52:55.233767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T11:53:03.003491+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac784459-dbff-42ed-941e-70c741392dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":92840425},{"type":"v1","addr":"172.21.15.100:6817","nonce":92840425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":92840425},{"type":"v1","addr":"172.21.15.100:6819","nonce":92840425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6822","nonce":92840425},{"type":"v1","addr":"172.21.15.100:6823","nonce":92840425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":92840425},{"type":"v1","addr":"172.21.15.100:6821","nonce":92840425}]},"public_addr":"172.21.15.100:6817/92840425","cluster_addr":"172.21.15.100:6819/92840425","heartbeat_back_addr":"172.21.15.100:6823/92840425","heartbeat_front_addr":"172.21.15.100:6821/92840425","state":["exists","up"]},{"osd":1,"uuid":"8d8d39e7-e8f1-499a-a01b-e7ba49844eae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":1385319817},{"type":"v1","addr":"172.21.15.100:6809","nonce":1385319817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":1385319817},{"type":"v1","addr":"172.21.15.100:6811","nonce":1385319817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":1385319817},{"type":"v1","addr":"172.21.15.100:6815","nonce":1385319817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":1385319817},{"type":"v1","addr":"172.21.15.100:6813","nonce":1385319817}]},"public_addr":"172.21.15.100:6809/1385319817","cluster_addr":"172.21.15.100:6811/1385319817","heartbeat_back_addr":"172.21.15.100:6815/1385319817","heartbeat_front_addr":"172.21.15.100:6813/1385319817","state":["exists","up"]},{"osd":2,"uuid":"e0bd3e30-0ff3-4067-83d3-ef8d84b29097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":2354314235},{"type":"v1","addr":"172.21.15.100:6801","nonce":2354314235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":2354314235},{"type":"v1","addr":"172.21.15.100:6803","nonce":2354314235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":2354314235},{"type":"v1","addr":"172.21.15.100:6807","nonce":2354314235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":2354314235},{"type":"v1","addr":"172.21.15.100:6805","nonce":2354314235}]},"public_addr":"172.21.15.100:6801/2354314235","cluster_addr":"172.21.15.100:6803/2354314235","heartbeat_back_addr":"172.21.15.100:6807/2354314235","heartbeat_front_addr":"172.21.15.100:6805/2354314235","state":["exists","up"]},{"osd":3,"uuid":"eca4429b-38cd-488a-b31d-c5856d701bde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3206511700},{"type":"v1","addr":"172.21.15.122:6809","nonce":3206511700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3206511700},{"type":"v1","addr":"172.21.15.122:6811","nonce":3206511700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3206511700},{"type":"v1","addr":"172.21.15.122:6815","nonce":3206511700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3206511700},{"type":"v1","addr":"172.21.15.122:6813","nonce":3206511700}]},"public_addr":"172.21.15.122:6809/3206511700","cluster_addr":"172.21.15.122:6811/3206511700","heartbeat_back_addr":"172.21.15.122:6815/3206511700","heartbeat_front_addr":"172.21.15.122:6813/3206511700","state":["exists","up"]},{"osd":4,"uuid":"b7ddc8f1-38d5-496a-9bf7-181eab7e5516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":777240702},{"type":"v1","addr":"172.21.15.122:6817","nonce":777240702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":777240702},{"type":"v1","addr":"172.21.15.122:6819","nonce":777240702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":777240702},{"type":"v1","addr":"172.21.15.122:6823","nonce":777240702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":777240702},{"type":"v1","addr":"172.21.15.122:6821","nonce":777240702}]},"public_addr":"172.21.15.122:6817/777240702","cluster_addr":"172.21.15.122:6819/777240702","heartbeat_back_addr":"172.21.15.122:6823/777240702","heartbeat_front_addr":"172.21.15.122:6821/777240702","state":["exists","up"]},{"osd":5,"uuid":"9010f8e0-c97b-451c-8b29-3ccb99e982a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":904725860},{"type":"v1","addr":"172.21.15.122:6801","nonce":904725860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":904725860},{"type":"v1","addr":"172.21.15.122:6803","nonce":904725860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":904725860},{"type":"v1","addr":"172.21.15.122:6807","nonce":904725860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":904725860},{"type":"v1","addr":"172.21.15.122:6805","nonce":904725860}]},"public_addr":"172.21.15.122:6801/904725860","cluster_addr":"172.21.15.122:6803/904725860","heartbeat_back_addr":"172.21.15.122:6807/904725860","heartbeat_front_addr":"172.21.15.122:6805/904725860","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:54.752725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:53.791663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:53.535397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:54.348980+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:54.658725+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:53.932256+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:53:07.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.513+0000 7f06334b8640 1 -- 172.21.15.100:0/3020730363 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f061003dfd0 msgr2=0x7f0610040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.513+0000 7f06334b8640 1 --2- 172.21.15.100:0/3020730363 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f061003dfd0 0x7f0610040490 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f06200043e0 tx=0x7f0620007ef0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.513+0000 7f06334b8640 10 monclient: shutdown 2024-09-13T11:53:07.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.513+0000 7f06334b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:07.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.513+0000 7f06334b8640 1 -- 172.21.15.100:0/3020730363 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f062c137a60 msgr2=0x7f062c136700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.513+0000 7f06334b8640 1 --2- 172.21.15.100:0/3020730363 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f062c137a60 0x7f062c136700 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0628031000 tx=0x7f06280046c0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.513+0000 7f06334b8640 1 -- 172.21.15.100:0/3020730363 shutdown_connections 2024-09-13T11:53:07.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.513+0000 7f06334b8640 1 --2- 172.21.15.100:0/3020730363 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f061003dfd0 0x7f0610040490 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.513+0000 7f06334b8640 1 --2- 172.21.15.100:0/3020730363 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f062c13ce90 0x7f062c09c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.513+0000 7f06334b8640 1 --2- 172.21.15.100:0/3020730363 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f062c137a60 0x7f062c136700 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.513+0000 7f06334b8640 1 --2- 172.21.15.100:0/3020730363 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f062c0a2980 0x7f062c09eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.513+0000 7f06334b8640 1 -- 172.21.15.100:0/3020730363 >> 172.21.15.100:0/3020730363 conn(0x7f062c0aa580 msgr2=0x7f062c1400a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:07.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.513+0000 7f06334b8640 1 -- 172.21.15.100:0/3020730363 shutdown_connections 2024-09-13T11:53:07.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.513+0000 7f06334b8640 1 -- 172.21.15.100:0/3020730363 wait complete. 2024-09-13T11:53:07.526 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-13T11:53:07.526 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-13T11:53:07.527 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-13T11:53:07.527 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-13T11:53:07.527 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-13T11:53:07.528 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-13T11:53:07.640 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.638+0000 7ff52359e640 10 monclient: get_monmap_and_config 2024-09-13T11:53:07.640 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.638+0000 7f3d90ba4640 10 monclient: get_monmap_and_config 2024-09-13T11:53:07.640 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.638+0000 7f3d90ba4640 10 monclient: build_initial_monmap 2024-09-13T11:53:07.640 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.638+0000 7f3d90ba4640 10 monclient: monmap: 2024-09-13T11:53:07.641 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:07.641 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.641 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:07.639629+0000 2024-09-13T11:53:07.641 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:07.639629+0000 2024-09-13T11:53:07.641 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:07.641 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.641 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:07.641 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:07.641 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:07.641 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:07.641 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:07.641 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.642 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.638+0000 7ff52359e640 10 monclient: build_initial_monmap 2024-09-13T11:53:07.642 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.638+0000 7ff52359e640 10 monclient: monmap: 2024-09-13T11:53:07.642 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:07.642 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.642 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:07.639629+0000 2024-09-13T11:53:07.642 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:07.639629+0000 2024-09-13T11:53:07.642 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:07.642 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.642 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:07.642 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:07.643 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:07.643 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:07.643 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:07.643 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.643 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.639+0000 7ff52359e640 1 Processor -- start 2024-09-13T11:53:07.643 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.639+0000 7f3d90ba4640 1 Processor -- start 2024-09-13T11:53:07.644 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.639+0000 7f3d90ba4640 1 -- start start 2024-09-13T11:53:07.644 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.639+0000 7ff52359e640 1 -- start start 2024-09-13T11:53:07.644 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.639+0000 7f3d90ba4640 10 monclient: init 2024-09-13T11:53:07.645 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.639+0000 7ff52359e640 10 monclient: init 2024-09-13T11:53:07.645 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d90ba4640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:07.645 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d90ba4640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-13T11:53:07.646 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d90ba4640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d8c137a60 0x7f3d8c0a2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.646 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d90ba4640 10 monclient(hunting): picked mon.noname-b con 0x7f3d8c137a60 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:07.646 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d90ba4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3d8c137e40 0x7f3d8c0a4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.646 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d90ba4640 10 monclient(hunting): picked mon.noname-c con 0x7f3d8c137e40 addr v2:172.21.15.122:3300/0 2024-09-13T11:53:07.646 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7ff52359e640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:07.647 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7ff52359e640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-13T11:53:07.647 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7ff52359e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff51c029430 0x7ff51c0270b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.647 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7ff52359e640 10 monclient(hunting): picked mon.noname-c con 0x7ff51c029430 addr v2:172.21.15.122:3300/0 2024-09-13T11:53:07.647 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7ff52359e640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff51c027ab0 0x7ff51c021850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.647 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d90ba4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3d8c138240 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:07.647 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d90ba4640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d8c0a5be0 con 0x7f3d8c138240 2024-09-13T11:53:07.647 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d90ba4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.647 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d90ba4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f3d8c0a4410 con 0x7f3d8c137e40 2024-09-13T11:53:07.647 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d90ba4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.647 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d90ba4640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f3d8c0a76b0 con 0x7f3d8c137a60 2024-09-13T11:53:07.647 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d90ba4640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:07.648 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d90ba4640 10 monclient(hunting): authenticate will time out at 768.233441s 2024-09-13T11:53:07.648 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7ff52359e640 10 monclient(hunting): picked mon.noname-b con 0x7ff51c027ab0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:07.648 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7ff52359e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff51c0275f0 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:07.648 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7ff521d9b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff51c027ab0 0x7ff51c021850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.648 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d8ad9d640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f3d8c138240 0x7f3d8c0a4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:57906/0 (socket says 172.21.15.100:57906) 2024-09-13T11:53:07.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d8ad9d640 1 -- 172.21.15.100:0/280325101 learned_addr learned my addr 172.21.15.100:0/280325101 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:53:07.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7ff521d9b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff51c027ab0 0x7ff51c021850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47254/0 (socket says 172.21.15.100:47254) 2024-09-13T11:53:07.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d83fff640 1 --2- 172.21.15.100:0/280325101 >> v2:172.21.15.122:3300/0 conn(0x7f3d8c137e40 0x7f3d8c0a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d83fff640 10 monclient(hunting): get_auth_request con 0x7f3d8c137e40 auth_method 0 2024-09-13T11:53:07.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d8a59c640 1 --2- 172.21.15.100:0/280325101 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d8c137a60 0x7f3d8c0a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d83fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d8a59c640 10 monclient(hunting): get_auth_request con 0x7f3d8c137a60 auth_method 0 2024-09-13T11:53:07.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d8a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d8a59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d8a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7ff521d9b640 1 -- 172.21.15.100:0/1977548624 learned_addr learned my addr 172.21.15.100:0/1977548624 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:07.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7f3d83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7f3d83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7f3d89d9b640 1 -- 172.21.15.100:0/280325101 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1994516919 0 0) 0x7f3d8c0a5be0 con 0x7f3d8c138240 2024-09-13T11:53:07.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7f3d83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7f3d89d9b640 10 monclient(hunting): _negotiate 2024-09-13T11:53:07.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7f3d89d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7f3d89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7f3d89d9b640 10 monclient(hunting): my global_id is 4263 2024-09-13T11:53:07.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7f3d89d9b640 1 -- 172.21.15.100:0/280325101 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d78003520 con 0x7f3d8c138240 2024-09-13T11:53:07.652 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7ff52259c640 1 --2- 172.21.15.100:0/1977548624 >> v2:172.21.15.122:3300/0 conn(0x7ff51c029430 0x7ff51c0270b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.652 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.640+0000 7ff52359e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff51c024460 con 0x7ff51c0275f0 2024-09-13T11:53:07.652 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7f3d8a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.652 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7f3d8a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.652 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7f3d8a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.652 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7f3d83fff640 10 monclient(hunting): handle_auth_done global_id 4246 payload 274 2024-09-13T11:53:07.653 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff52359e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.653 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7f3d83fff640 1 -- 172.21.15.100:0/280325101 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d8c137a60 msgr2=0x7f3d8c0a2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:07.654 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff52359e640 1 -- 172.21.15.100:0/1977548624 --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7ff51c022ee0 con 0x7ff51c029430 2024-09-13T11:53:07.654 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff52359e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.654 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff52359e640 1 -- 172.21.15.100:0/1977548624 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7ff51c029810 con 0x7ff51c027ab0 2024-09-13T11:53:07.655 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff52359e640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:07.655 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff52359e640 10 monclient(hunting): authenticate will time out at 768.234093s 2024-09-13T11:53:07.655 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff521d9b640 10 monclient(hunting): get_auth_request con 0x7ff51c027ab0 auth_method 0 2024-09-13T11:53:07.655 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff521d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.655 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff521d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.655 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff521d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.655 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff52259c640 10 monclient(hunting): get_auth_request con 0x7ff51c029430 auth_method 0 2024-09-13T11:53:07.655 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff52259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.655 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff52259c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.655 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff52259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.655 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff521d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.655 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff521d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff521d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff52159a640 1 -- 172.21.15.100:0/1977548624 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 947867618 0 0) 0x7ff52406e720 con 0x7ff51c0275f0 2024-09-13T11:53:07.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff52159a640 10 monclient(hunting): _negotiate 2024-09-13T11:53:07.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff52159a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff52159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff52159a640 10 monclient(hunting): my global_id is 4249 2024-09-13T11:53:07.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff52159a640 1 -- 172.21.15.100:0/1977548624 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff50c003520 con 0x7ff51c0275f0 2024-09-13T11:53:07.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff52259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff52259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff52259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff521d9b640 10 monclient(hunting): handle_auth_done global_id 4229 payload 274 2024-09-13T11:53:07.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff521d9b640 1 -- 172.21.15.100:0/1977548624 >> v2:172.21.15.122:3300/0 conn(0x7ff51c029430 msgr2=0x7ff51c0270b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:07.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff521d9b640 1 --2- 172.21.15.100:0/1977548624 >> v2:172.21.15.122:3300/0 conn(0x7ff51c029430 0x7ff51c0270b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff521d9b640 1 -- 172.21.15.100:0/1977548624 >> v1:172.21.15.122:6789/0 conn(0x7ff51c0275f0 legacy=0x7ff51c021d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff521d9b640 10 monclient: _finish_hunting 0 2024-09-13T11:53:07.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff521d9b640 1 monclient: found mon.noname-b 2024-09-13T11:53:07.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff521d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:07.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff521d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:53:07.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7ff521d9b640 1 -- 172.21.15.100:0/1977548624 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff51c023ea0 con 0x7ff51c027ab0 2024-09-13T11:53:07.658 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7f3d83fff640 1 --2- 172.21.15.100:0/280325101 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d8c137a60 0x7f3d8c0a2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.658 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7f3d83fff640 1 -- 172.21.15.100:0/280325101 >> v1:172.21.15.100:6789/0 conn(0x7f3d8c138240 legacy=0x7f3d8c0a4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.658 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7f3d83fff640 10 monclient: _finish_hunting 0 2024-09-13T11:53:07.658 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7f3d83fff640 1 monclient: found mon.noname-c 2024-09-13T11:53:07.658 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7f3d83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:07.658 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7f3d83fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.122:3300/0 2024-09-13T11:53:07.658 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.641+0000 7f3d83fff640 1 -- 172.21.15.100:0/280325101 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d8c0a19e0 con 0x7f3d8c137e40 2024-09-13T11:53:07.658 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7f3d83fff640 1 --2- 172.21.15.100:0/280325101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d8c137e40 0x7f3d8c0a4950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c0a7880 tx=0x7f3d7402a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=800dd9f49a8b6d12 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.659 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7ff521d9b640 1 --2- 172.21.15.100:0/1977548624 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff51c027ab0 0x7ff51c021850 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff518009870 tx=0x7ff51802a1d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=435679b0b745ca30 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.659 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7ff52159a640 1 -- 172.21.15.100:0/1977548624 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff518037070 con 0x7ff51c027ab0 2024-09-13T11:53:07.659 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7ff52159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.659 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7ff52159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:07.659 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7ff52159a640 10 monclient: dump: 2024-09-13T11:53:07.659 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.659 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.659 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.660 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.660 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.660 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.660 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.660 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.660 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.660 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.661 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7f3d89d9b640 1 -- 172.21.15.100:0/280325101 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d74037070 con 0x7f3d8c137e40 2024-09-13T11:53:07.661 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7f3d89d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.661 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7ff52159a640 10 monclient: _finish_auth 0 2024-09-13T11:53:07.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7ff52159a640 10 monclient: _check_auth_tickets 2024-09-13T11:53:07.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7ff52159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:07.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7ff52159a640 1 -- 172.21.15.100:0/1977548624 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff518032440 con 0x7ff51c027ab0 2024-09-13T11:53:07.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7ff52159a640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:07.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7ff52159a640 1 -- 172.21.15.100:0/1977548624 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff518032810 con 0x7ff51c027ab0 2024-09-13T11:53:07.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7ff52159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7ff52159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:07.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7ff52159a640 10 monclient: dump: 2024-09-13T11:53:07.664 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.664 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.665 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.665 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.665 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.665 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.665 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.665 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.665 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.666 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7f3d89d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:07.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7f3d89d9b640 10 monclient: dump: 2024-09-13T11:53:07.666 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.666 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.666 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.666 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.666 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.666 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.667 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.667 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.667 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.667 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7f3d89d9b640 10 monclient: _finish_auth 0 2024-09-13T11:53:07.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7f3d89d9b640 10 monclient: _check_auth_tickets 2024-09-13T11:53:07.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7f3d89d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:07.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7f3d89d9b640 1 -- 172.21.15.100:0/280325101 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d7402adb0 con 0x7f3d8c137e40 2024-09-13T11:53:07.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7f3d89d9b640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:07.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7f3d89d9b640 1 -- 172.21.15.100:0/280325101 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d74032660 con 0x7f3d8c137e40 2024-09-13T11:53:07.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7ff52359e640 5 monclient: authenticate success, global_id 4229 2024-09-13T11:53:07.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7f3d89d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7f3d89d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:53:07.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7f3d89d9b640 10 monclient: dump: 2024-09-13T11:53:07.668 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.668 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.669 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.669 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.669 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.669 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.669 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.669 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.669 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.669 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7f3d90ba4640 5 monclient: authenticate success, global_id 4246 2024-09-13T11:53:07.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7f3d90ba4640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:07.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7f3d90ba4640 4 set_mon_vals no callback set 2024-09-13T11:53:07.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7ff52359e640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:07.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.642+0000 7ff52359e640 4 set_mon_vals no callback set 2024-09-13T11:53:07.671 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.643+0000 7f3d90ba4640 10 monclient: shutdown 2024-09-13T11:53:07.671 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.643+0000 7f3d90ba4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:07.671 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.643+0000 7f3d90ba4640 1 -- 172.21.15.100:0/280325101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d8c137e40 msgr2=0x7f3d8c0a4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.671 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.643+0000 7f3d90ba4640 1 --2- 172.21.15.100:0/280325101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d8c137e40 0x7f3d8c0a4950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c0a7880 tx=0x7f3d7402a060 comp rx=0 tx=0).stop 2024-09-13T11:53:07.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.643+0000 7ff52359e640 10 monclient: shutdown 2024-09-13T11:53:07.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.643+0000 7ff52359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:07.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.643+0000 7f3d90ba4640 1 -- 172.21.15.100:0/280325101 shutdown_connections 2024-09-13T11:53:07.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.643+0000 7f3d90ba4640 1 --2- 172.21.15.100:0/280325101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d8c137e40 0x7f3d8c0a4950 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.643+0000 7f3d90ba4640 1 --2- 172.21.15.100:0/280325101 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d8c137a60 0x7f3d8c0a2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.643+0000 7f3d90ba4640 1 -- 172.21.15.100:0/280325101 >> 172.21.15.100:0/280325101 conn(0x7f3d8c0aa580 msgr2=0x7f3d8c0a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:07.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.643+0000 7ff52359e640 1 -- 172.21.15.100:0/1977548624 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff51c027ab0 msgr2=0x7ff51c021850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.643+0000 7ff52359e640 1 --2- 172.21.15.100:0/1977548624 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff51c027ab0 0x7ff51c021850 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff518009870 tx=0x7ff51802a1d0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.643+0000 7f3d90ba4640 1 -- 172.21.15.100:0/280325101 shutdown_connections 2024-09-13T11:53:07.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.643+0000 7f3d90ba4640 1 -- 172.21.15.100:0/280325101 wait complete. 2024-09-13T11:53:07.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.643+0000 7ff52359e640 1 -- 172.21.15.100:0/1977548624 shutdown_connections 2024-09-13T11:53:07.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.643+0000 7ff52359e640 1 --2- 172.21.15.100:0/1977548624 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff51c027ab0 0x7ff51c021850 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.643+0000 7ff52359e640 1 --2- 172.21.15.100:0/1977548624 >> v2:172.21.15.122:3300/0 conn(0x7ff51c029430 0x7ff51c0270b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.643+0000 7ff52359e640 1 -- 172.21.15.100:0/1977548624 >> 172.21.15.100:0/1977548624 conn(0x7ff51c006340 msgr2=0x7ff51c006750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:07.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.643+0000 7ff52359e640 1 -- 172.21.15.100:0/1977548624 shutdown_connections 2024-09-13T11:53:07.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.643+0000 7ff52359e640 1 -- 172.21.15.100:0/1977548624 wait complete. 2024-09-13T11:53:07.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.644+0000 7ff52359e640 10 monclient: build_initial_monmap 2024-09-13T11:53:07.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.644+0000 7f3d90ba4640 10 monclient: build_initial_monmap 2024-09-13T11:53:07.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.644+0000 7ff52359e640 10 monclient: monmap: 2024-09-13T11:53:07.675 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:07.675 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:07.675 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:07.675 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:07.675 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:07.675 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.675 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:07.675 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:07.676 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:07.676 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.644+0000 7f3d90ba4640 10 monclient: monmap: 2024-09-13T11:53:07.677 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:07.677 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:07.677 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:07.677 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:07.677 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:07.677 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.678 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:07.678 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:07.678 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:07.678 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.645+0000 7f3d90ba4640 1 Processor -- start 2024-09-13T11:53:07.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.645+0000 7ff52359e640 1 Processor -- start 2024-09-13T11:53:07.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.645+0000 7f3d90ba4640 1 -- start start 2024-09-13T11:53:07.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.645+0000 7ff52359e640 1 -- start start 2024-09-13T11:53:07.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.645+0000 7f3d90ba4640 10 monclient: init 2024-09-13T11:53:07.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.645+0000 7ff52359e640 10 monclient: init 2024-09-13T11:53:07.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.646+0000 7f3d90ba4640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:07.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.646+0000 7ff52359e640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:07.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.646+0000 7f3d90ba4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:53:07.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.646+0000 7ff52359e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:53:07.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.646+0000 7f3d90ba4640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d8c137a60 0x7f3d8c13c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.646+0000 7ff52359e640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff51c0275f0 0x7ff51c1506f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.646+0000 7ff52359e640 10 monclient(hunting): picked mon.noname-a con 0x7ff51c0275f0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:07.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.646+0000 7ff52359e640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff51c027ab0 0x7ff51c0ba6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.646+0000 7f3d90ba4640 10 monclient(hunting): picked mon.noname-c con 0x7f3d8c137a60 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:07.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.646+0000 7f3d90ba4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d8c137e40 0x7f3d8c13c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.646+0000 7f3d8a59c640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d8c137a60 0x7f3d8c13c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.646+0000 7f3d90ba4640 10 monclient(hunting): picked mon.noname-b con 0x7f3d8c137e40 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:07.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.646+0000 7f3d8a59c640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d8c137a60 0x7f3d8c13c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47276/0 (socket says 172.21.15.100:47276) 2024-09-13T11:53:07.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.646+0000 7f3d8a59c640 1 -- 172.21.15.100:0/3756484886 learned_addr learned my addr 172.21.15.100:0/3756484886 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:07.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.646+0000 7f3d90ba4640 1 --2- 172.21.15.100:0/3756484886 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d8c138240 0x7f3d8c13cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.646+0000 7ff52259c640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff51c0275f0 0x7ff51c1506f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.646+0000 7ff52259c640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff51c0275f0 0x7ff51c1506f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49454/0 (socket says 172.21.15.100:49454) 2024-09-13T11:53:07.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.646+0000 7ff521d9b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff51c027ab0 0x7ff51c0ba6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.646+0000 7ff52259c640 1 -- 172.21.15.100:0/4042006767 learned_addr learned my addr 172.21.15.100:0/4042006767 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:07.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.646+0000 7ff52359e640 10 monclient(hunting): picked mon.noname-c con 0x7ff51c027ab0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:07.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.646+0000 7ff52359e640 1 --2- 172.21.15.100:0/4042006767 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff51c029430 0x7ff51c0a8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff52359e640 10 monclient(hunting): picked mon.noname-b con 0x7ff51c029430 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:07.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff52359e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.687 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7f3d83fff640 1 --2- 172.21.15.100:0/3756484886 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d8c137e40 0x7f3d8c13c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.687 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7f3d8ad9d640 1 --2- 172.21.15.100:0/3756484886 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d8c138240 0x7f3d8c13cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff52359e640 1 -- 172.21.15.100:0/4042006767 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7ff51c0bbb90 con 0x7ff51c0275f0 2024-09-13T11:53:07.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff522d9d640 1 --2- 172.21.15.100:0/4042006767 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff51c029430 0x7ff51c0a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff52359e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff52359e640 1 -- 172.21.15.100:0/4042006767 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff51c0bba10 con 0x7ff51c029430 2024-09-13T11:53:07.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff52359e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff52359e640 1 -- 172.21.15.100:0/4042006767 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7ff51c0224a0 con 0x7ff51c027ab0 2024-09-13T11:53:07.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff52359e640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:07.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff52359e640 10 monclient(hunting): authenticate will time out at 768.240337s 2024-09-13T11:53:07.689 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7f3d90ba4640 10 monclient(hunting): picked mon.noname-a con 0x7f3d8c138240 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:07.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7f3d90ba4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7f3d90ba4640 1 -- 172.21.15.100:0/3756484886 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f3d8c06ff00 con 0x7f3d8c138240 2024-09-13T11:53:07.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7f3d90ba4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7f3d90ba4640 1 -- 172.21.15.100:0/3756484886 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f3d8c06fd80 con 0x7f3d8c137e40 2024-09-13T11:53:07.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7f3d90ba4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7f3d90ba4640 1 -- 172.21.15.100:0/3756484886 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f3d8c0a55a0 con 0x7f3d8c137a60 2024-09-13T11:53:07.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7f3d90ba4640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:07.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff52259c640 10 monclient(hunting): get_auth_request con 0x7ff51c0275f0 auth_method 0 2024-09-13T11:53:07.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff52259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff52259c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff52259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff521d9b640 10 monclient(hunting): get_auth_request con 0x7ff51c027ab0 auth_method 0 2024-09-13T11:53:07.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff521d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff521d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff521d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff521d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff521d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff521d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff522d9d640 10 monclient(hunting): get_auth_request con 0x7ff51c029430 auth_method 0 2024-09-13T11:53:07.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff522d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff522d9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff522d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff52259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff52259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff52259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff521d9b640 10 monclient(hunting): handle_auth_done global_id 4232 payload 824 2024-09-13T11:53:07.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff521d9b640 1 -- 172.21.15.100:0/4042006767 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff51c029430 msgr2=0x7ff51c0a8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.693 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff521d9b640 1 --2- 172.21.15.100:0/4042006767 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff51c029430 0x7ff51c0a8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.693 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff521d9b640 1 -- 172.21.15.100:0/4042006767 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff51c0275f0 msgr2=0x7ff51c1506f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.693 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff521d9b640 1 --2- 172.21.15.100:0/4042006767 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff51c0275f0 0x7ff51c1506f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.693 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7f3d90ba4640 10 monclient(hunting): authenticate will time out at 768.240910s 2024-09-13T11:53:07.694 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff521d9b640 10 monclient: _finish_hunting 0 2024-09-13T11:53:07.694 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff521d9b640 1 monclient: found mon.noname-c 2024-09-13T11:53:07.694 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff521d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:07.694 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff521d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:53:07.694 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7ff521d9b640 1 -- 172.21.15.100:0/4042006767 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff51c02d2a0 con 0x7ff51c027ab0 2024-09-13T11:53:07.694 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7ff52259c640 1 --2- 172.21.15.100:0/4042006767 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff51c0275f0 0x7ff51c1506f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:07.695 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7f3d8a59c640 10 monclient(hunting): get_auth_request con 0x7f3d8c137a60 auth_method 0 2024-09-13T11:53:07.695 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7f3d8a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.695 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7f3d8a59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.695 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7f3d8a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.695 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7f3d83fff640 10 monclient(hunting): get_auth_request con 0x7f3d8c137e40 auth_method 0 2024-09-13T11:53:07.695 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7f3d83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.696 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7f3d83fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.696 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.647+0000 7f3d83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.696 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d8ad9d640 10 monclient(hunting): get_auth_request con 0x7f3d8c138240 auth_method 0 2024-09-13T11:53:07.696 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d8ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.696 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d8ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.696 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d8ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.697 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d8a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.697 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d8a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.697 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d8a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.697 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d8ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.697 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d8ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.697 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d8ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.697 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.698 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7ff521d9b640 1 --2- 172.21.15.100:0/4042006767 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff51c027ab0 0x7ff51c0ba6f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff518031000 tx=0x7ff5180046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.698 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d8ad9d640 10 monclient(hunting): handle_auth_done global_id 4269 payload 824 2024-09-13T11:53:07.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d8ad9d640 1 -- 172.21.15.100:0/3756484886 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d8c137a60 msgr2=0x7f3d8c13c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d8ad9d640 1 --2- 172.21.15.100:0/3756484886 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d8c137a60 0x7f3d8c13c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d8ad9d640 1 -- 172.21.15.100:0/3756484886 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d8c137e40 msgr2=0x7f3d8c13c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d8ad9d640 1 --2- 172.21.15.100:0/3756484886 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d8c137e40 0x7f3d8c13c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d8ad9d640 10 monclient: _finish_hunting 0 2024-09-13T11:53:07.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d8ad9d640 1 monclient: found mon.noname-a 2024-09-13T11:53:07.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d8ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:07.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d8ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:07.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d8ad9d640 1 -- 172.21.15.100:0/3756484886 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d8c120cc0 con 0x7f3d8c138240 2024-09-13T11:53:07.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d8a59c640 1 --2- 172.21.15.100:0/3756484886 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d8c137a60 0x7f3d8c13c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:07.700 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7ff512ffd640 1 -- 172.21.15.100:0/4042006767 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff518041020 con 0x7ff51c027ab0 2024-09-13T11:53:07.700 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7ff512ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.700 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7ff512ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:07.700 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7ff512ffd640 10 monclient: dump: 2024-09-13T11:53:07.700 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.700 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.700 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.700 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.700 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.701 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.701 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.701 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.701 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.701 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7ff512ffd640 10 monclient: _finish_auth 0 2024-09-13T11:53:07.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7ff512ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:53:07.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7ff512ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:07.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7f3d8ad9d640 1 --2- 172.21.15.100:0/3756484886 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d8c138240 0x7f3d8c13cca0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3d8400ca80 tx=0x7f3d8400cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7ff512ffd640 1 -- 172.21.15.100:0/4042006767 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff51803d640 con 0x7ff51c027ab0 2024-09-13T11:53:07.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7ff512ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:07.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7ff512ffd640 1 -- 172.21.15.100:0/4042006767 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff518037040 con 0x7ff51c027ab0 2024-09-13T11:53:07.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7ff512ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7ff512ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:07.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7ff512ffd640 10 monclient: dump: 2024-09-13T11:53:07.702 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.702 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.702 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.702 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.702 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.703 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.703 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.703 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.703 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.703 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7ff52359e640 5 monclient: authenticate success, global_id 4232 2024-09-13T11:53:07.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7ff52359e640 10 monclient: _renew_subs 2024-09-13T11:53:07.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7ff52359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:07.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7ff52359e640 1 -- 172.21.15.100:0/4042006767 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff51c150da0 con 0x7ff51c027ab0 2024-09-13T11:53:07.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.648+0000 7ff520d99640 4 set_mon_vals no callback set 2024-09-13T11:53:07.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.649+0000 7ff52359e640 10 monclient: _renew_subs 2024-09-13T11:53:07.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.649+0000 7ff52359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:07.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.649+0000 7ff52359e640 1 -- 172.21.15.100:0/4042006767 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff51c0abec0 con 0x7ff51c027ab0 2024-09-13T11:53:07.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.649+0000 7f3d82ffd640 1 -- 172.21.15.100:0/3756484886 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d84017070 con 0x7f3d8c138240 2024-09-13T11:53:07.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.649+0000 7f3d82ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.649+0000 7f3d82ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:07.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.649+0000 7f3d82ffd640 10 monclient: dump: 2024-09-13T11:53:07.705 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.705 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.705 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.705 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.705 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.705 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.705 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.705 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.705 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.705 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.649+0000 7f3d82ffd640 10 monclient: _finish_auth 0 2024-09-13T11:53:07.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.649+0000 7f3d82ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:53:07.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.649+0000 7f3d82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:07.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.649+0000 7f3d82ffd640 1 -- 172.21.15.100:0/3756484886 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d8400ac30 con 0x7f3d8c138240 2024-09-13T11:53:07.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.649+0000 7f3d82ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:07.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.649+0000 7f3d82ffd640 1 -- 172.21.15.100:0/3756484886 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d84012400 con 0x7f3d8c138240 2024-09-13T11:53:07.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.649+0000 7f3d82ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.649+0000 7f3d82ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:07.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.649+0000 7f3d82ffd640 10 monclient: dump: 2024-09-13T11:53:07.706 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.706 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.706 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.706 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.706 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.706 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.706 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.707 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.707 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.707 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.649+0000 7f3d8959a640 4 set_mon_vals no callback set 2024-09-13T11:53:07.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.649+0000 7f3d90ba4640 5 monclient: authenticate success, global_id 4269 2024-09-13T11:53:07.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.649+0000 7f3d90ba4640 10 monclient: _renew_subs 2024-09-13T11:53:07.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.649+0000 7f3d90ba4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:07.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.649+0000 7f3d90ba4640 1 -- 172.21.15.100:0/3756484886 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d8c120740 con 0x7f3d8c138240 2024-09-13T11:53:07.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.649+0000 7f3d90ba4640 10 monclient: _renew_subs 2024-09-13T11:53:07.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.649+0000 7f3d90ba4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:07.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.649+0000 7f3d90ba4640 1 -- 172.21.15.100:0/3756484886 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d8c13e3c0 con 0x7f3d8c138240 2024-09-13T11:53:07.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.650+0000 7ff512ffd640 1 -- 172.21.15.100:0/4042006767 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7ff518044020 con 0x7ff51c027ab0 2024-09-13T11:53:07.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.650+0000 7f3d82ffd640 1 -- 172.21.15.100:0/3756484886 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f3d8401a070 con 0x7f3d8c138240 2024-09-13T11:53:07.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.650+0000 7f3d82ffd640 1 --2- 172.21.15.100:0/3756484886 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f3d6c03dff0 0x7f3d6c0404b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.651+0000 7f3d90ba4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:07.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.651+0000 7f3d90ba4640 1 -- 172.21.15.100:0/3756484886 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3d8c0a8780 con 0x7f3d8c138240 2024-09-13T11:53:07.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.651+0000 7f3d82ffd640 1 -- 172.21.15.100:0/3756484886 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7f3d840551c0 con 0x7f3d8c138240 2024-09-13T11:53:07.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.651+0000 7f3d82ffd640 1 --2- 172.21.15.100:0/3756484886 >> [v2:172.21.15.100:6808/1385319817,v1:172.21.15.100:6809/1385319817] conn(0x7f3d6c043e60 0x7f3d6c0462a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.651+0000 7f3d82ffd640 1 -- 172.21.15.100:0/3756484886 --> [v2:172.21.15.100:6808/1385319817,v1:172.21.15.100:6809/1385319817] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d6c046970 con 0x7f3d6c043e60 2024-09-13T11:53:07.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.651+0000 7ff512ffd640 1 --2- 172.21.15.100:0/4042006767 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff50403dff0 0x7ff5040404b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.651+0000 7ff52359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:07.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.651+0000 7ff52359e640 1 -- 172.21.15.100:0/4042006767 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff51c0270b0 con 0x7ff51c027ab0 2024-09-13T11:53:07.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.651+0000 7f3d83fff640 1 --2- 172.21.15.100:0/3756484886 >> [v2:172.21.15.100:6808/1385319817,v1:172.21.15.100:6809/1385319817] conn(0x7f3d6c043e60 0x7f3d6c0462a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.651+0000 7f3d83fff640 10 monclient: get_auth_request con 0x7f3d6c043e60 auth_method 0 2024-09-13T11:53:07.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.651+0000 7ff512ffd640 1 -- 172.21.15.100:0/4042006767 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7ff518073050 con 0x7ff51c027ab0 2024-09-13T11:53:07.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.651+0000 7ff512ffd640 1 --2- 172.21.15.100:0/4042006767 >> [v2:172.21.15.122:6816/777240702,v1:172.21.15.122:6817/777240702] conn(0x7ff504043e60 0x7ff5040462a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.651+0000 7ff512ffd640 1 -- 172.21.15.100:0/4042006767 --> [v2:172.21.15.122:6816/777240702,v1:172.21.15.122:6817/777240702] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff504046970 con 0x7ff504043e60 2024-09-13T11:53:07.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.651+0000 7f3d8a59c640 1 --2- 172.21.15.100:0/3756484886 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f3d6c03dff0 0x7f3d6c0404b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.651+0000 7f3d8a59c640 10 monclient: get_auth_request con 0x7f3d6c03dff0 auth_method 0 2024-09-13T11:53:07.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.651+0000 7ff512ffd640 1 -- 172.21.15.100:0/4042006767 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7ff51803bcc0 con 0x7ff51c027ab0 2024-09-13T11:53:07.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.651+0000 7ff512ffd640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-13T11:53:07.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.651+0000 7f3d82ffd640 1 -- 172.21.15.100:0/3756484886 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f3d84039050 con 0x7f3d8c138240 2024-09-13T11:53:07.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.651+0000 7f3d82ffd640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-13T11:53:07.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.651+0000 7ff522d9d640 1 --2- 172.21.15.100:0/4042006767 >> [v2:172.21.15.122:6816/777240702,v1:172.21.15.122:6817/777240702] conn(0x7ff504043e60 0x7ff5040462a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.714 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.651+0000 7ff522d9d640 10 monclient: get_auth_request con 0x7ff504043e60 auth_method 0 2024-09-13T11:53:07.714 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.651+0000 7ff52259c640 1 --2- 172.21.15.100:0/4042006767 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff50403dff0 0x7ff5040404b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.714 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.651+0000 7ff52259c640 10 monclient: get_auth_request con 0x7ff50403dff0 auth_method 0 2024-09-13T11:53:07.714 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.651+0000 7f3d83fff640 1 --2- 172.21.15.100:0/3756484886 >> [v2:172.21.15.100:6808/1385319817,v1:172.21.15.100:6809/1385319817] conn(0x7f3d6c043e60 0x7f3d6c0462a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.714 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.652+0000 7ff522d9d640 1 --2- 172.21.15.100:0/4042006767 >> [v2:172.21.15.122:6816/777240702,v1:172.21.15.122:6817/777240702] conn(0x7ff504043e60 0x7ff5040462a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.714 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.652+0000 7ff52259c640 1 --2- 172.21.15.100:0/4042006767 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff50403dff0 0x7ff5040404b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff51c0ab330 tx=0x7ff514018040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.652+0000 7f3d8a59c640 1 --2- 172.21.15.100:0/3756484886 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f3d6c03dff0 0x7f3d6c0404b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c0112c0 tx=0x7f3d7c009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.652+0000 7f3d82ffd640 1 -- 172.21.15.100:0/3756484886 <== osd.1 v2:172.21.15.100:6808/1385319817 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3d6c046970 con 0x7f3d6c043e60 2024-09-13T11:53:07.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.652+0000 7ff512ffd640 1 -- 172.21.15.100:0/4042006767 <== osd.4 v2:172.21.15.122:6816/777240702 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff504046970 con 0x7ff504043e60 2024-09-13T11:53:07.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.666+0000 7f3d90ba4640 1 -- 172.21.15.100:0/3756484886 --> [v2:172.21.15.100:6808/1385319817,v1:172.21.15.100:6809/1385319817] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3d8c0ab040 con 0x7f3d6c043e60 2024-09-13T11:53:07.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.666+0000 7ff52359e640 1 -- 172.21.15.100:0/4042006767 --> [v2:172.21.15.122:6816/777240702,v1:172.21.15.122:6817/777240702] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff51c0bac30 con 0x7ff504043e60 2024-09-13T11:53:07.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.666+0000 7f3d82ffd640 1 -- 172.21.15.100:0/3756484886 <== osd.1 v2:172.21.15.100:6808/1385319817 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3d8c0ab040 con 0x7f3d6c043e60 2024-09-13T11:53:07.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.666+0000 7f3d90ba4640 1 -- 172.21.15.100:0/3756484886 >> [v2:172.21.15.100:6808/1385319817,v1:172.21.15.100:6809/1385319817] conn(0x7f3d6c043e60 msgr2=0x7f3d6c0462a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.666+0000 7f3d90ba4640 1 --2- 172.21.15.100:0/3756484886 >> [v2:172.21.15.100:6808/1385319817,v1:172.21.15.100:6809/1385319817] conn(0x7f3d6c043e60 0x7f3d6c0462a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.666+0000 7ff512ffd640 1 -- 172.21.15.100:0/4042006767 <== osd.4 v2:172.21.15.122:6816/777240702 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff51c0bac30 con 0x7ff504043e60 2024-09-13T11:53:07.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.667+0000 7f3d90ba4640 1 -- 172.21.15.100:0/3756484886 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f3d6c03dff0 msgr2=0x7f3d6c0404b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.667+0000 7f3d90ba4640 1 --2- 172.21.15.100:0/3756484886 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f3d6c03dff0 0x7f3d6c0404b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c0112c0 tx=0x7f3d7c009190 comp rx=0 tx=0).stop 2024-09-13T11:53:07.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.667+0000 7f3d90ba4640 10 monclient: shutdown 2024-09-13T11:53:07.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.667+0000 7f3d90ba4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:07.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.667+0000 7f3d90ba4640 1 -- 172.21.15.100:0/3756484886 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d8c138240 msgr2=0x7f3d8c13cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.667+0000 7f3d90ba4640 1 --2- 172.21.15.100:0/3756484886 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d8c138240 0x7f3d8c13cca0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3d8400ca80 tx=0x7f3d8400cf40 comp rx=0 tx=0).stop 2024-09-13T11:53:07.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.667+0000 7f3d8ad9d640 1 -- 172.21.15.100:0/3756484886 reap_dead start 2024-09-13T11:53:07.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.667+0000 7ff52359e640 1 -- 172.21.15.100:0/4042006767 >> [v2:172.21.15.122:6816/777240702,v1:172.21.15.122:6817/777240702] conn(0x7ff504043e60 msgr2=0x7ff5040462a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.667+0000 7ff52359e640 1 --2- 172.21.15.100:0/4042006767 >> [v2:172.21.15.122:6816/777240702,v1:172.21.15.122:6817/777240702] conn(0x7ff504043e60 0x7ff5040462a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.667+0000 7f3d90ba4640 1 -- 172.21.15.100:0/3756484886 shutdown_connections 2024-09-13T11:53:07.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.667+0000 7f3d90ba4640 1 -- 172.21.15.100:0/3756484886 >> 172.21.15.100:0/3756484886 conn(0x7f3d8c0aa580 msgr2=0x7f3d8c0a02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:07.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.667+0000 7f3d90ba4640 1 -- 172.21.15.100:0/3756484886 shutdown_connections 2024-09-13T11:53:07.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.667+0000 7ff52359e640 1 -- 172.21.15.100:0/4042006767 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff50403dff0 msgr2=0x7ff5040404b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.667+0000 7ff52359e640 1 --2- 172.21.15.100:0/4042006767 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff50403dff0 0x7ff5040404b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff51c0ab330 tx=0x7ff514018040 comp rx=0 tx=0).stop 2024-09-13T11:53:07.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.667+0000 7ff52359e640 10 monclient: shutdown 2024-09-13T11:53:07.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.667+0000 7ff52359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:07.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.667+0000 7ff52359e640 1 -- 172.21.15.100:0/4042006767 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff51c027ab0 msgr2=0x7ff51c0ba6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.667+0000 7ff52359e640 1 --2- 172.21.15.100:0/4042006767 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff51c027ab0 0x7ff51c0ba6f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff518031000 tx=0x7ff5180046c0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.667+0000 7ff522d9d640 1 -- 172.21.15.100:0/4042006767 reap_dead start 2024-09-13T11:53:07.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.667+0000 7f3d90ba4640 1 -- 172.21.15.100:0/3756484886 wait complete. 2024-09-13T11:53:07.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.667+0000 7ff52359e640 1 -- 172.21.15.100:0/4042006767 shutdown_connections 2024-09-13T11:53:07.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.667+0000 7ff52359e640 1 -- 172.21.15.100:0/4042006767 >> 172.21.15.100:0/4042006767 conn(0x7ff51c006340 msgr2=0x7ff51c025090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:07.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.667+0000 7ff52359e640 1 -- 172.21.15.100:0/4042006767 shutdown_connections 2024-09-13T11:53:07.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.667+0000 7ff52359e640 1 -- 172.21.15.100:0/4042006767 wait complete. 2024-09-13T11:53:07.724 INFO:teuthology.orchestra.run.smithi100.stdout:47244640262 2024-09-13T11:53:07.724 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T11:53:07.725 INFO:teuthology.orchestra.run.smithi100.stdout:51539607557 2024-09-13T11:53:07.725 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T11:53:07.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.682+0000 7f08f33a4640 10 monclient: get_monmap_and_config 2024-09-13T11:53:07.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.682+0000 7f08f33a4640 10 monclient: build_initial_monmap 2024-09-13T11:53:07.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.682+0000 7f08f33a4640 10 monclient: monmap: 2024-09-13T11:53:07.726 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:07.726 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.726 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:07.683344+0000 2024-09-13T11:53:07.727 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:07.683344+0000 2024-09-13T11:53:07.727 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:07.727 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.727 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:07.727 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:07.727 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:07.727 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:07.727 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:07.727 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.683+0000 7f08f33a4640 1 Processor -- start 2024-09-13T11:53:07.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.683+0000 7f08f33a4640 1 -- start start 2024-09-13T11:53:07.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.683+0000 7f08f33a4640 10 monclient: init 2024-09-13T11:53:07.728 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.683+0000 7f9c5e356640 10 monclient: get_monmap_and_config 2024-09-13T11:53:07.728 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.683+0000 7f9c5e356640 10 monclient: build_initial_monmap 2024-09-13T11:53:07.728 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.683+0000 7f9c5e356640 10 monclient: monmap: 2024-09-13T11:53:07.728 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:07.729 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.729 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:07.684536+0000 2024-09-13T11:53:07.729 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:07.684536+0000 2024-09-13T11:53:07.729 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:07.729 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.729 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:07.729 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:07.730 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:07.730 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:07.730 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:07.730 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.730 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.683+0000 7f7105b64640 10 monclient: get_monmap_and_config 2024-09-13T11:53:07.730 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.683+0000 7f7105b64640 10 monclient: build_initial_monmap 2024-09-13T11:53:07.730 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.683+0000 7f7105b64640 10 monclient: monmap: 2024-09-13T11:53:07.730 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:07.730 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.731 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:07.684596+0000 2024-09-13T11:53:07.731 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:07.684596+0000 2024-09-13T11:53:07.731 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:07.731 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.731 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:07.731 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:07.731 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:07.731 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:07.731 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:07.731 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.683+0000 7fa07f5f3640 10 monclient: get_monmap_and_config 2024-09-13T11:53:07.732 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.683+0000 7fa07f5f3640 10 monclient: build_initial_monmap 2024-09-13T11:53:07.732 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.683+0000 7fa07f5f3640 10 monclient: monmap: 2024-09-13T11:53:07.732 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:07.732 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.732 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:07.684608+0000 2024-09-13T11:53:07.732 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:07.684608+0000 2024-09-13T11:53:07.732 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:07.732 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.732 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:07.732 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:07.732 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:07.732 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:07.733 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:07.733 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.683+0000 7f08f33a4640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:07.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.683+0000 7f08f33a4640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-13T11:53:07.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.683+0000 7f08f33a4640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08ec06ea50 0x7f08ec0a2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.683+0000 7f08f33a4640 10 monclient(hunting): picked mon.noname-b con 0x7f08ec06ea50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:07.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.683+0000 7f08f33a4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f08ec06ee30 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:07.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.683+0000 7f08f33a4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f08ec13c9f0 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:07.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.683+0000 7f08f33a4640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08ec0a1590 con 0x7f08ec06ee30 2024-09-13T11:53:07.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.683+0000 7f08f33a4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08ec0a1870 con 0x7f08ec13c9f0 2024-09-13T11:53:07.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.683+0000 7f08f33a4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.683+0000 7f08f33a4640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f08ec0a1b50 con 0x7f08ec06ea50 2024-09-13T11:53:07.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.683+0000 7f08f33a4640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:07.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.683+0000 7f08f33a4640 10 monclient(hunting): authenticate will time out at 768.276915s 2024-09-13T11:53:07.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f08f1140640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08ec06ea50 0x7f08ec0a2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f08f093f640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f08ec06ee30 0x7f08ec06f250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:57920/0 (socket says 172.21.15.100:57920) 2024-09-13T11:53:07.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f08f093f640 1 -- 172.21.15.100:0/4243497370 learned_addr learned my addr 172.21.15.100:0/4243497370 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:53:07.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f08f1140640 10 monclient(hunting): get_auth_request con 0x7f08ec06ea50 auth_method 0 2024-09-13T11:53:07.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f08f1140640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f08f1140640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f08f1140640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f08f1140640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f08f1140640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f08f1140640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f08e3fff640 1 -- 172.21.15.100:0/4243497370 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 696873745 0 0) 0x7f08ec0a1590 con 0x7f08ec06ee30 2024-09-13T11:53:07.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f08e3fff640 10 monclient(hunting): _negotiate 2024-09-13T11:53:07.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f08e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f08e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f08e3fff640 10 monclient(hunting): my global_id is 4272 2024-09-13T11:53:07.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f08e3fff640 1 -- 172.21.15.100:0/4243497370 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08d8003520 con 0x7f08ec06ee30 2024-09-13T11:53:07.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f08f1140640 10 monclient(hunting): handle_auth_done global_id 4238 payload 274 2024-09-13T11:53:07.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f08f1140640 1 -- 172.21.15.100:0/4243497370 >> v1:172.21.15.122:6789/0 conn(0x7f08ec13c9f0 legacy=0x7f08ec13ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7fa07f5f3640 1 Processor -- start 2024-09-13T11:53:07.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f08f1140640 1 -- 172.21.15.100:0/4243497370 >> v1:172.21.15.100:6789/0 conn(0x7f08ec06ee30 legacy=0x7f08ec06f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f9c5e356640 1 Processor -- start 2024-09-13T11:53:07.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f08f1140640 10 monclient: _finish_hunting 0 2024-09-13T11:53:07.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f08f1140640 1 monclient: found mon.noname-b 2024-09-13T11:53:07.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f08f1140640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:07.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f08f1140640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:53:07.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f08f1140640 1 -- 172.21.15.100:0/4243497370 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08ec06fca0 con 0x7f08ec06ea50 2024-09-13T11:53:07.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f9c5e356640 1 -- start start 2024-09-13T11:53:07.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f9c5e356640 10 monclient: init 2024-09-13T11:53:07.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7fa07f5f3640 1 -- start start 2024-09-13T11:53:07.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7fa07f5f3640 10 monclient: init 2024-09-13T11:53:07.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.684+0000 7f7105b64640 1 Processor -- start 2024-09-13T11:53:07.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f7105b64640 1 -- start start 2024-09-13T11:53:07.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f7105b64640 10 monclient: init 2024-09-13T11:53:07.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa07f5f3640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:07.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa07f5f3640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-13T11:53:07.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa07f5f3640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa078073270 0x7fa078073650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa07f5f3640 10 monclient(hunting): picked mon.noname-b con 0x7fa078073270 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:07.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c5e356640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:07.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c5e356640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-13T11:53:07.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa07f5f3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa078073b90 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:07.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c5e356640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9c5806ea50 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:07.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c5e356640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9c580a2860 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:07.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c5e356640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c5813da00 0x7f9c5813fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c5e356640 10 monclient(hunting): picked mon.noname-b con 0x7f9c5813da00 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:07.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c5e356640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c580a1590 con 0x7f9c580a2860 2024-09-13T11:53:07.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c5e356640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c580a1870 con 0x7f9c5806ea50 2024-09-13T11:53:07.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c5e356640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c5e356640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f9c580a2cb0 con 0x7f9c5813da00 2024-09-13T11:53:07.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c5e356640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:07.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c5e356640 10 monclient(hunting): authenticate will time out at 768.278468s 2024-09-13T11:53:07.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa07f5f3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa07809fb90 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:07.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa07f5f3640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0780a10b0 con 0x7fa07809fb90 2024-09-13T11:53:07.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa07f5f3640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0780a4680 con 0x7fa078073b90 2024-09-13T11:53:07.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa07f5f3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c5c8f3640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c5813da00 0x7f9c5813fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c5c8f3640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c5813da00 0x7f9c5813fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47298/0 (socket says 172.21.15.100:47298) 2024-09-13T11:53:07.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c5c8f3640 1 -- 172.21.15.100:0/3111096953 learned_addr learned my addr 172.21.15.100:0/3111096953 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:07.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c5c8f3640 10 monclient(hunting): get_auth_request con 0x7f9c5813da00 auth_method 0 2024-09-13T11:53:07.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c5c8f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c5c8f3640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c5c8f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c577fe640 1 --1- 172.21.15.100:0/3111096953 >> v1:172.21.15.100:6789/0 conn(0x7f9c580a2860 0x7f9c5806f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:57928/0 (socket says 172.21.15.100:57928) 2024-09-13T11:53:07.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c5c8f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c5c8f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa07f5f3640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fa0780a19e0 con 0x7fa078073270 2024-09-13T11:53:07.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa07f5f3640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:07.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa07f5f3640 10 monclient(hunting): authenticate will time out at 768.278417s 2024-09-13T11:53:07.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c5c8f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.746 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa07db90640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fa07809fb90 0x7fa0780a3f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:57940/0 (socket says 172.21.15.100:57940) 2024-09-13T11:53:07.746 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa07db90640 1 -- 172.21.15.100:0/2942708018 learned_addr learned my addr 172.21.15.100:0/2942708018 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:53:07.746 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa07d38f640 1 --2- 172.21.15.100:0/2942708018 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa078073270 0x7fa078073650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.746 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa07d38f640 10 monclient(hunting): get_auth_request con 0x7fa078073270 auth_method 0 2024-09-13T11:53:07.746 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa07d38f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.746 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa07d38f640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.746 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa07d38f640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.746 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa07d38f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.746 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa07d38f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.746 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa07d38f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.747 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f08f1140640 1 --2- 172.21.15.100:0/4243497370 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08ec06ea50 0x7f08ec0a2860 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f08e80097d0 tx=0x7f08e802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3219fe25c8b91d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.747 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c5c8f3640 10 monclient(hunting): handle_auth_done global_id 4241 payload 274 2024-09-13T11:53:07.747 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa06ffff640 1 -- 172.21.15.100:0/2942708018 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2393118612 0 0) 0x7fa074001490 con 0x7fa07809fb90 2024-09-13T11:53:07.747 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c5c8f3640 1 -- 172.21.15.100:0/3111096953 >> v1:172.21.15.122:6789/0 conn(0x7f9c5806ea50 legacy=0x7f9c5806ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.747 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7f9c56ffd640 1 -- 172.21.15.100:0/3111096953 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1560979504 0 0) 0x7f9c580a1590 con 0x7f9c580a2860 2024-09-13T11:53:07.748 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa06ffff640 10 monclient(hunting): _negotiate 2024-09-13T11:53:07.748 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa06ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.748 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa06ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.748 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.685+0000 7fa06ffff640 10 monclient(hunting): my global_id is 4275 2024-09-13T11:53:07.748 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7fa06ffff640 1 -- 172.21.15.100:0/2942708018 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa064003520 con 0x7fa07809fb90 2024-09-13T11:53:07.748 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f9c5c8f3640 1 -- 172.21.15.100:0/3111096953 >> v1:172.21.15.100:6789/0 conn(0x7f9c580a2860 legacy=0x7f9c5806f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.749 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f9c5c8f3640 10 monclient: _finish_hunting 0 2024-09-13T11:53:07.749 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f9c5c8f3640 1 monclient: found mon.noname-b 2024-09-13T11:53:07.749 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f9c5c8f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:07.749 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f9c5c8f3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:53:07.749 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f9c5c8f3640 1 -- 172.21.15.100:0/3111096953 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c580a1ba0 con 0x7f9c5813da00 2024-09-13T11:53:07.749 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f9c56ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:53:07.749 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7fa07d38f640 10 monclient(hunting): handle_auth_done global_id 4244 payload 274 2024-09-13T11:53:07.750 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7fa07d38f640 1 -- 172.21.15.100:0/2942708018 >> v1:172.21.15.122:6789/0 conn(0x7fa078073b90 legacy=0x7fa0780a3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.750 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7fa07d38f640 1 -- 172.21.15.100:0/2942708018 >> v1:172.21.15.100:6789/0 conn(0x7fa07809fb90 legacy=0x7fa0780a3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.750 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7fa07d38f640 10 monclient: _finish_hunting 0 2024-09-13T11:53:07.750 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7fa07d38f640 1 monclient: found mon.noname-b 2024-09-13T11:53:07.750 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7fa07d38f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:07.750 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7fa07d38f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:53:07.750 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7fa07d38f640 1 -- 172.21.15.100:0/2942708018 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0780a1390 con 0x7fa078073270 2024-09-13T11:53:07.750 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f7105b64640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:07.751 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f08e3fff640 1 -- 172.21.15.100:0/4243497370 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e8037070 con 0x7f08ec06ea50 2024-09-13T11:53:07.751 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f08e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.751 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f08e3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:07.751 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f08e3fff640 10 monclient: dump: 2024-09-13T11:53:07.751 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.751 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.751 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.751 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.751 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.752 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.752 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.752 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.752 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.752 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.752 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f08e3fff640 10 monclient: _finish_auth 0 2024-09-13T11:53:07.752 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f08e3fff640 10 monclient: _check_auth_tickets 2024-09-13T11:53:07.752 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f08e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:07.752 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f08e3fff640 1 -- 172.21.15.100:0/4243497370 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08e802ae50 con 0x7f08ec06ea50 2024-09-13T11:53:07.752 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f08e3fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:07.752 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f08e3fff640 1 -- 172.21.15.100:0/4243497370 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e8032660 con 0x7f08ec06ea50 2024-09-13T11:53:07.752 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f08e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.752 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f08e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:07.753 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f08e3fff640 10 monclient: dump: 2024-09-13T11:53:07.753 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.753 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.753 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.753 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.753 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.753 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.753 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.753 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.753 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.753 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.753 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f08f33a4640 5 monclient: authenticate success, global_id 4238 2024-09-13T11:53:07.753 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f08f33a4640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:07.753 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f08f33a4640 4 set_mon_vals no callback set 2024-09-13T11:53:07.754 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f7105b64640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-13T11:53:07.754 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f7105b64640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7100131d90 0x7f71000a23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f7105b64640 10 monclient(hunting): picked mon.noname-b con 0x7f7100131d90 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:07.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f7105b64640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7100132170 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:07.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f7105b64640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f710013a820 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:07.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f7105b64640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f710009fea0 con 0x7f710013a820 2024-09-13T11:53:07.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f7105b64640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71000a0180 con 0x7f7100132170 2024-09-13T11:53:07.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f7105b64640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f7105b64640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f71000a0460 con 0x7f7100131d90 2024-09-13T11:53:07.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f7105b64640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:07.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f7105b64640 10 monclient(hunting): authenticate will time out at 768.279329s 2024-09-13T11:53:07.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f70ff7fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7100131d90 0x7f71000a23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f70ff7fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7100131d90 0x7f71000a23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47306/0 (socket says 172.21.15.100:47306) 2024-09-13T11:53:07.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f70ff7fe640 1 -- 172.21.15.100:0/2813406295 learned_addr learned my addr 172.21.15.100:0/2813406295 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:07.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f70ff7fe640 10 monclient(hunting): get_auth_request con 0x7f7100131d90 auth_method 0 2024-09-13T11:53:07.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f70ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f70ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f70ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f70ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f70ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f70ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f70ff7fe640 10 monclient(hunting): handle_auth_done global_id 4247 payload 274 2024-09-13T11:53:07.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f70ff7fe640 1 -- 172.21.15.100:0/2813406295 >> v1:172.21.15.122:6789/0 conn(0x7f7100132170 legacy=0x7f7100132550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f70ff7fe640 1 -- 172.21.15.100:0/2813406295 >> v1:172.21.15.100:6789/0 conn(0x7f710013a820 legacy=0x7f710013cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f70ff7fe640 10 monclient: _finish_hunting 0 2024-09-13T11:53:07.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f70ff7fe640 1 monclient: found mon.noname-b 2024-09-13T11:53:07.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f70ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:07.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f70ff7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:53:07.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f70ff7fe640 1 -- 172.21.15.100:0/2813406295 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f710009caf0 con 0x7f7100131d90 2024-09-13T11:53:07.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f70fe7fc640 1 -- 172.21.15.100:0/2813406295 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3684222678 0 0) 0x7f710009fea0 con 0x7f710013a820 2024-09-13T11:53:07.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f70fe7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:53:07.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.686+0000 7f9c5c8f3640 1 --2- 172.21.15.100:0/3111096953 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c5813da00 0x7f9c5813fdf0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9c480098a0 tx=0x7f9c4802a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b87780c8798e011 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f9c56ffd640 1 -- 172.21.15.100:0/3111096953 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c48037070 con 0x7f9c5813da00 2024-09-13T11:53:07.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f9c56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f9c56ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:07.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f9c56ffd640 10 monclient: dump: 2024-09-13T11:53:07.758 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.758 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.758 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.758 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.758 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.758 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.758 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.758 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.758 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.758 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.758 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f9c56ffd640 10 monclient: _finish_auth 0 2024-09-13T11:53:07.758 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f9c56ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:53:07.758 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f9c56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:07.758 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f9c5e356640 5 monclient: authenticate success, global_id 4241 2024-09-13T11:53:07.758 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f9c5e356640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:07.759 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7fa07d38f640 1 --2- 172.21.15.100:0/2942708018 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa078073270 0x7fa078073650 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa0700097d0 tx=0x7fa07002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53f051113cf4c042 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.759 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f9c56ffd640 1 -- 172.21.15.100:0/3111096953 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c48032440 con 0x7f9c5813da00 2024-09-13T11:53:07.759 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f9c56ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:07.759 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f9c56ffd640 1 -- 172.21.15.100:0/3111096953 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c48032810 con 0x7f9c5813da00 2024-09-13T11:53:07.759 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f9c5e356640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:07.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f9c5e356640 4 set_mon_vals no callback set 2024-09-13T11:53:07.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f9c56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f9c56ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:07.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f08f33a4640 10 monclient: shutdown 2024-09-13T11:53:07.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f9c56ffd640 10 monclient: dump: 2024-09-13T11:53:07.761 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.761 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.761 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.762 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.762 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.762 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.762 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.762 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.762 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.762 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f08f33a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:07.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f08f33a4640 1 -- 172.21.15.100:0/4243497370 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08ec06ea50 msgr2=0x7f08ec0a2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f08f33a4640 1 --2- 172.21.15.100:0/4243497370 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08ec06ea50 0x7f08ec0a2860 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f08e80097d0 tx=0x7f08e802a100 comp rx=0 tx=0).stop 2024-09-13T11:53:07.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7fa06ffff640 1 -- 172.21.15.100:0/2942708018 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa070037070 con 0x7fa078073270 2024-09-13T11:53:07.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7fa06ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7fa06ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:07.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7fa06ffff640 10 monclient: dump: 2024-09-13T11:53:07.764 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.765 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.765 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.765 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.765 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.765 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.765 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.765 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.766 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.766 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7fa06ffff640 10 monclient: _finish_auth 0 2024-09-13T11:53:07.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7fa06ffff640 10 monclient: _check_auth_tickets 2024-09-13T11:53:07.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7fa06ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:07.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7fa07f5f3640 5 monclient: authenticate success, global_id 4244 2024-09-13T11:53:07.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7fa07f5f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:07.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7fa06ffff640 1 -- 172.21.15.100:0/2942708018 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa070032440 con 0x7fa078073270 2024-09-13T11:53:07.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7fa06ffff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:07.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7fa07f5f3640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:07.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7fa07f5f3640 4 set_mon_vals no callback set 2024-09-13T11:53:07.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f08f33a4640 1 -- 172.21.15.100:0/4243497370 shutdown_connections 2024-09-13T11:53:07.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f08f33a4640 1 --2- 172.21.15.100:0/4243497370 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08ec06ea50 0x7f08ec0a2860 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f08f33a4640 1 -- 172.21.15.100:0/4243497370 >> 172.21.15.100:0/4243497370 conn(0x7f08ec0a68e0 msgr2=0x7f08ec0a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:07.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7fa06ffff640 1 -- 172.21.15.100:0/2942708018 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0700327f0 con 0x7fa078073270 2024-09-13T11:53:07.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7fa06ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7fa06ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:07.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7fa06ffff640 10 monclient: dump: 2024-09-13T11:53:07.768 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.768 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.768 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.768 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.769 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.769 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.769 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.769 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.769 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.769 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f08f33a4640 1 -- 172.21.15.100:0/4243497370 shutdown_connections 2024-09-13T11:53:07.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f08f33a4640 1 -- 172.21.15.100:0/4243497370 wait complete. 2024-09-13T11:53:07.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f70ff7fe640 1 --2- 172.21.15.100:0/2813406295 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7100131d90 0x7f71000a23d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f70f000dcc0 tx=0x7f70f002e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b8c21948bb30d1c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f70fe7fc640 1 -- 172.21.15.100:0/2813406295 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f003a070 con 0x7f7100131d90 2024-09-13T11:53:07.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f70fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f70fe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:07.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.687+0000 7f70fe7fc640 10 monclient: dump: 2024-09-13T11:53:07.772 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.772 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.773 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.773 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.773 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.773 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.773 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.773 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.774 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.774 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f70fe7fc640 10 monclient: _finish_auth 0 2024-09-13T11:53:07.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f70fe7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:53:07.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f70fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:07.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f7105b64640 5 monclient: authenticate success, global_id 4247 2024-09-13T11:53:07.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f9c5e356640 10 monclient: shutdown 2024-09-13T11:53:07.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f70fe7fc640 1 -- 172.21.15.100:0/2813406295 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70f000b650 con 0x7f7100131d90 2024-09-13T11:53:07.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f7105b64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:07.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f70fe7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:07.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f7105b64640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:07.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f7105b64640 4 set_mon_vals no callback set 2024-09-13T11:53:07.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f70fe7fc640 1 -- 172.21.15.100:0/2813406295 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f000ba20 con 0x7f7100131d90 2024-09-13T11:53:07.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f70fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.777 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f70fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:07.777 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f70fe7fc640 10 monclient: dump: 2024-09-13T11:53:07.777 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.777 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.777 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.777 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.778 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.778 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.778 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.778 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.778 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.778 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f9c5e356640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:07.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f9c5e356640 1 -- 172.21.15.100:0/3111096953 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c5813da00 msgr2=0x7f9c5813fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f9c5e356640 1 --2- 172.21.15.100:0/3111096953 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c5813da00 0x7f9c5813fdf0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9c480098a0 tx=0x7f9c4802a1c0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f9c5e356640 1 -- 172.21.15.100:0/3111096953 shutdown_connections 2024-09-13T11:53:07.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f9c5e356640 1 --2- 172.21.15.100:0/3111096953 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c5813da00 0x7f9c5813fdf0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f9c5e356640 1 -- 172.21.15.100:0/3111096953 >> 172.21.15.100:0/3111096953 conn(0x7f9c580a68e0 msgr2=0x7f9c580a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:07.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f9c5e356640 1 -- 172.21.15.100:0/3111096953 shutdown_connections 2024-09-13T11:53:07.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f9c5e356640 1 -- 172.21.15.100:0/3111096953 wait complete. 2024-09-13T11:53:07.782 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7fa07f5f3640 10 monclient: shutdown 2024-09-13T11:53:07.782 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7fa07f5f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:07.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7fa07f5f3640 1 -- 172.21.15.100:0/2942708018 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa078073270 msgr2=0x7fa078073650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7fa07f5f3640 1 --2- 172.21.15.100:0/2942708018 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa078073270 0x7fa078073650 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa0700097d0 tx=0x7fa07002a180 comp rx=0 tx=0).stop 2024-09-13T11:53:07.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7fa07f5f3640 1 -- 172.21.15.100:0/2942708018 shutdown_connections 2024-09-13T11:53:07.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7fa07f5f3640 1 --2- 172.21.15.100:0/2942708018 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa078073270 0x7fa078073650 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7fa07f5f3640 1 -- 172.21.15.100:0/2942708018 >> 172.21.15.100:0/2942708018 conn(0x7fa0780a8660 msgr2=0x7fa0780a4950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:07.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f7105b64640 10 monclient: shutdown 2024-09-13T11:53:07.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f7105b64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:07.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f7105b64640 1 -- 172.21.15.100:0/2813406295 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7100131d90 msgr2=0x7f71000a23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f7105b64640 1 --2- 172.21.15.100:0/2813406295 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7100131d90 0x7f71000a23d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f70f000dcc0 tx=0x7f70f002e4b0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f08f33a4640 10 monclient: build_initial_monmap 2024-09-13T11:53:07.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f08f33a4640 10 monclient: monmap: 2024-09-13T11:53:07.786 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:07.786 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:07.786 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:07.786 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:07.786 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:07.786 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.787 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:07.787 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:07.787 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:07.787 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f7105b64640 1 -- 172.21.15.100:0/2813406295 shutdown_connections 2024-09-13T11:53:07.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f7105b64640 1 --2- 172.21.15.100:0/2813406295 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7100131d90 0x7f71000a23d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7f7105b64640 1 -- 172.21.15.100:0/2813406295 >> 172.21.15.100:0/2813406295 conn(0x7f71000a2c90 msgr2=0x7f71000a30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:07.789 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7fa07f5f3640 1 -- 172.21.15.100:0/2942708018 shutdown_connections 2024-09-13T11:53:07.789 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.688+0000 7fa07f5f3640 1 -- 172.21.15.100:0/2942708018 wait complete. 2024-09-13T11:53:07.790 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.689+0000 7f7105b64640 1 -- 172.21.15.100:0/2813406295 shutdown_connections 2024-09-13T11:53:07.790 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.689+0000 7f7105b64640 1 -- 172.21.15.100:0/2813406295 wait complete. 2024-09-13T11:53:07.791 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.689+0000 7f9c5e356640 10 monclient: build_initial_monmap 2024-09-13T11:53:07.791 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.689+0000 7f9c5e356640 10 monclient: monmap: 2024-09-13T11:53:07.791 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:07.791 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:07.791 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:07.791 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:07.792 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:07.792 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.792 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:07.792 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:07.792 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:07.792 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.793 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.689+0000 7f08f33a4640 1 Processor -- start 2024-09-13T11:53:07.793 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.689+0000 7fa07f5f3640 10 monclient: build_initial_monmap 2024-09-13T11:53:07.793 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.689+0000 7fa07f5f3640 10 monclient: monmap: 2024-09-13T11:53:07.794 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:07.794 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:07.794 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:07.794 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:07.794 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:07.794 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.794 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:07.795 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:07.795 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:07.795 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.795 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.689+0000 7f08f33a4640 1 -- start start 2024-09-13T11:53:07.795 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.689+0000 7f08f33a4640 10 monclient: init 2024-09-13T11:53:07.796 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.689+0000 7f7105b64640 10 monclient: build_initial_monmap 2024-09-13T11:53:07.796 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.689+0000 7f7105b64640 10 monclient: monmap: 2024-09-13T11:53:07.796 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:07.796 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:07.796 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:07.796 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:07.796 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:07.796 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.796 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:07.796 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:07.797 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:07.797 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.797 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.689+0000 7fa07f5f3640 1 Processor -- start 2024-09-13T11:53:07.798 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.689+0000 7fa07f5f3640 1 -- start start 2024-09-13T11:53:07.798 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.689+0000 7fa07f5f3640 10 monclient: init 2024-09-13T11:53:07.798 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.689+0000 7f9c5e356640 1 Processor -- start 2024-09-13T11:53:07.799 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7f7105b64640 1 Processor -- start 2024-09-13T11:53:07.799 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7f7105b64640 1 -- start start 2024-09-13T11:53:07.799 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07f5f3640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:07.799 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07f5f3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:53:07.799 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07f5f3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa078073270 0x7fa0781d0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.799 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07f5f3640 10 monclient(hunting): picked mon.noname-b con 0x7fa078073270 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:07.799 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07f5f3640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa078073b90 0x7fa07813d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.800 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07f5f3640 10 monclient(hunting): picked mon.noname-a con 0x7fa078073b90 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:07.800 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7f7105b64640 10 monclient: init 2024-09-13T11:53:07.800 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07f5f3640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa07809fb90 0x7fa07813dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.800 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07f5f3640 10 monclient(hunting): picked mon.noname-c con 0x7fa07809fb90 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:07.800 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07f5f3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.800 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07f5f3640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fa0780a04d0 con 0x7fa078073b90 2024-09-13T11:53:07.800 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07f5f3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.800 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07f5f3640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa0780a0350 con 0x7fa078073270 2024-09-13T11:53:07.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07f5f3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07f5f3640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fa07813e670 con 0x7fa07809fb90 2024-09-13T11:53:07.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07f5f3640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:07.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07f5f3640 10 monclient(hunting): authenticate will time out at 768.283256s 2024-09-13T11:53:07.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7f9c5e356640 1 -- start start 2024-09-13T11:53:07.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7f9c5e356640 10 monclient: init 2024-09-13T11:53:07.802 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7f08f33a4640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:07.802 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7f08f33a4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:53:07.802 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7f08f33a4640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08ec06ea50 0x7f08ec136e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.802 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7f08f33a4640 10 monclient(hunting): picked mon.noname-c con 0x7f08ec06ea50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:07.802 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7f08f33a4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08ec06ee30 0x7f08ec137370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.803 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07db90640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa07809fb90 0x7fa07813dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.803 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07cb8e640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa078073b90 0x7fa07813d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.803 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07cb8e640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa078073b90 0x7fa07813d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49482/0 (socket says 172.21.15.100:49482) 2024-09-13T11:53:07.803 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07db90640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa07809fb90 0x7fa07813dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47310/0 (socket says 172.21.15.100:47310) 2024-09-13T11:53:07.803 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07cb8e640 1 -- 172.21.15.100:0/2425434581 learned_addr learned my addr 172.21.15.100:0/2425434581 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:07.803 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7f08f33a4640 10 monclient(hunting): picked mon.noname-b con 0x7f08ec06ee30 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:07.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07cb8e640 10 monclient(hunting): get_auth_request con 0x7fa078073b90 auth_method 0 2024-09-13T11:53:07.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07cb8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07cb8e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07cb8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07db90640 10 monclient(hunting): get_auth_request con 0x7fa07809fb90 auth_method 0 2024-09-13T11:53:07.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07db90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07db90640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07db90640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07cb8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07cb8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7f08f33a4640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08ec13c9f0 0x7f08ec09da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7f08f33a4640 10 monclient(hunting): picked mon.noname-a con 0x7f08ec13c9f0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:07.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7f08f1140640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08ec06ea50 0x7f08ec136e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7f08f1140640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08ec06ea50 0x7f08ec136e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47318/0 (socket says 172.21.15.100:47318) 2024-09-13T11:53:07.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7f08f1140640 1 -- 172.21.15.100:0/3012792809 learned_addr learned my addr 172.21.15.100:0/3012792809 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:07.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07cb8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07db90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07db90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07db90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7f08f093f640 1 --2- 172.21.15.100:0/3012792809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08ec06ee30 0x7f08ec137370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07d38f640 1 --2- 172.21.15.100:0/2425434581 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa078073270 0x7fa0781d0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07d38f640 10 monclient(hunting): get_auth_request con 0x7fa078073270 auth_method 0 2024-09-13T11:53:07.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07d38f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07d38f640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07d38f640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07d38f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07d38f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07d38f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07db90640 10 monclient(hunting): handle_auth_done global_id 4250 payload 824 2024-09-13T11:53:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07db90640 1 -- 172.21.15.100:0/2425434581 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa078073270 msgr2=0x7fa0781d0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07db90640 1 --2- 172.21.15.100:0/2425434581 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa078073270 0x7fa0781d0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07db90640 1 -- 172.21.15.100:0/2425434581 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa078073b90 msgr2=0x7fa07813d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07db90640 1 --2- 172.21.15.100:0/2425434581 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa078073b90 0x7fa07813d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07db90640 10 monclient: _finish_hunting 0 2024-09-13T11:53:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07db90640 1 monclient: found mon.noname-c 2024-09-13T11:53:07.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07db90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:07.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07db90640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:53:07.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07db90640 1 -- 172.21.15.100:0/2425434581 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa07812ce10 con 0x7fa07809fb90 2024-09-13T11:53:07.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.690+0000 7fa07cb8e640 1 --2- 172.21.15.100:0/2425434581 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa078073b90 0x7fa07813d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:07.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7fa07db90640 1 --2- 172.21.15.100:0/2425434581 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa07809fb90 0x7fa07813dc50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa07400ca50 tx=0x7fa07400cf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f1941640 1 --2- 172.21.15.100:0/3012792809 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08ec13c9f0 0x7f08ec09da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f33a4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7fa06dffb640 1 -- 172.21.15.100:0/2425434581 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07400abb0 con 0x7fa07809fb90 2024-09-13T11:53:07.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7fa06dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c5e356640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:07.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7fa06dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:07.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7fa06dffb640 10 monclient: dump: 2024-09-13T11:53:07.810 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.810 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.811 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.811 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.811 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.811 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.811 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.811 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.811 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.811 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7fa06dffb640 10 monclient: _finish_auth 0 2024-09-13T11:53:07.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7fa06dffb640 10 monclient: _check_auth_tickets 2024-09-13T11:53:07.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7fa06dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:07.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f33a4640 1 -- 172.21.15.100:0/3012792809 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec0a5be0 con 0x7f08ec13c9f0 2024-09-13T11:53:07.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f33a4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f7105b64640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:07.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7fa06dffb640 1 -- 172.21.15.100:0/2425434581 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa07400ad50 con 0x7fa07809fb90 2024-09-13T11:53:07.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7fa06dffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:07.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7fa06dffb640 1 -- 172.21.15.100:0/2425434581 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa074012600 con 0x7fa07809fb90 2024-09-13T11:53:07.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7fa06dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7fa06dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:07.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7fa06dffb640 10 monclient: dump: 2024-09-13T11:53:07.813 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.813 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.813 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.814 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.814 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.814 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.814 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.814 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.814 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.814 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c5e356640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:53:07.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c5e356640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c5806ea50 0x7f9c58137420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f33a4640 1 -- 172.21.15.100:0/3012792809 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec0a5a60 con 0x7f08ec06ee30 2024-09-13T11:53:07.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f33a4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f7105b64640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:53:07.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7fa06f7fe640 4 set_mon_vals no callback set 2024-09-13T11:53:07.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c5e356640 10 monclient(hunting): picked mon.noname-b con 0x7f9c5806ea50 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:07.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f33a4640 1 -- 172.21.15.100:0/3012792809 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f08ec140500 con 0x7f08ec06ea50 2024-09-13T11:53:07.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f33a4640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:07.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f7105b64640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7100131d90 0x7f71001c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f7105b64640 10 monclient(hunting): picked mon.noname-a con 0x7f7100131d90 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:07.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7fa07f5f3640 5 monclient: authenticate success, global_id 4250 2024-09-13T11:53:07.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7fa07f5f3640 10 monclient: _renew_subs 2024-09-13T11:53:07.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7fa07f5f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:07.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7fa07f5f3640 1 -- 172.21.15.100:0/2425434581 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa07812d0f0 con 0x7fa07809fb90 2024-09-13T11:53:07.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c5e356640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9c580a2860 0x7f9c58137960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c5e356640 10 monclient(hunting): picked mon.noname-a con 0x7f9c580a2860 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:07.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f33a4640 10 monclient(hunting): authenticate will time out at 768.284324s 2024-09-13T11:53:07.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f7105b64640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7100132170 0x7f71000745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c5e356640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c5813da00 0x7f9c5809da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c5e356640 10 monclient(hunting): picked mon.noname-c con 0x7f9c5813da00 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:07.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f70ff7fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7100131d90 0x7f71001c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f70ff7fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7100131d90 0x7f71001c5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49500/0 (socket says 172.21.15.100:49500) 2024-09-13T11:53:07.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c57fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c5806ea50 0x7f9c58137420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c57fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c5806ea50 0x7f9c58137420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.100:50394/0 (socket says 172.21.15.100:50394) 2024-09-13T11:53:07.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c57fff640 1 -- 172.21.15.100:0/1088148368 learned_addr learned my addr 172.21.15.100:0/1088148368 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:07.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c5e356640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c5e356640 1 -- 172.21.15.100:0/1088148368 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f9c581404b0 con 0x7f9c580a2860 2024-09-13T11:53:07.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c5e356640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c5e356640 1 -- 172.21.15.100:0/1088148368 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9c58140330 con 0x7f9c5806ea50 2024-09-13T11:53:07.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c5e356640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c5e356640 1 -- 172.21.15.100:0/1088148368 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f9c58140630 con 0x7f9c5813da00 2024-09-13T11:53:07.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c5e356640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:07.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c5e356640 10 monclient(hunting): authenticate will time out at 768.284671s 2024-09-13T11:53:07.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c57fff640 10 monclient(hunting): get_auth_request con 0x7f9c5806ea50 auth_method 0 2024-09-13T11:53:07.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c57fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c5c8f3640 1 --2- 172.21.15.100:0/1088148368 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c5813da00 0x7f9c5809da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c5c8f3640 10 monclient(hunting): get_auth_request con 0x7f9c5813da00 auth_method 0 2024-09-13T11:53:07.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c5c8f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c5c8f3640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c5c8f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c577fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9c580a2860 0x7f9c58137960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c577fe640 10 monclient(hunting): get_auth_request con 0x7f9c580a2860 auth_method 0 2024-09-13T11:53:07.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c577fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7fa07f5f3640 10 monclient: _renew_subs 2024-09-13T11:53:07.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f1140640 10 monclient(hunting): get_auth_request con 0x7f08ec06ea50 auth_method 0 2024-09-13T11:53:07.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f1140640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f1140640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f1140640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f093f640 10 monclient(hunting): get_auth_request con 0x7f08ec06ee30 auth_method 0 2024-09-13T11:53:07.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f093f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f093f640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7fa07f5f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:07.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7fa07f5f3640 1 -- 172.21.15.100:0/2425434581 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa07812c060 con 0x7fa07809fb90 2024-09-13T11:53:07.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f093f640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f1140640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f1140640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f1140640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f093f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f093f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f093f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f093f640 10 monclient(hunting): handle_auth_done global_id 4273 payload 824 2024-09-13T11:53:07.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f093f640 1 -- 172.21.15.100:0/3012792809 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08ec06ea50 msgr2=0x7f08ec136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f9c5c8f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f9c5c8f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f9c5c8f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f9c57fff640 10 monclient(hunting): handle_auth_done global_id 4276 payload 824 2024-09-13T11:53:07.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f9c57fff640 1 -- 172.21.15.100:0/1088148368 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c5813da00 msgr2=0x7f9c5809da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f9c57fff640 1 --2- 172.21.15.100:0/1088148368 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c5813da00 0x7f9c5809da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f9c57fff640 1 -- 172.21.15.100:0/1088148368 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9c580a2860 msgr2=0x7f9c58137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f9c57fff640 1 --2- 172.21.15.100:0/1088148368 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9c580a2860 0x7f9c58137960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f9c57fff640 10 monclient: _finish_hunting 0 2024-09-13T11:53:07.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f9c57fff640 1 monclient: found mon.noname-b 2024-09-13T11:53:07.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f9c57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:07.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f9c57fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.122:3300/0 2024-09-13T11:53:07.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f9c57fff640 1 -- 172.21.15.100:0/1088148368 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c58120960 con 0x7f9c5806ea50 2024-09-13T11:53:07.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f093f640 1 --2- 172.21.15.100:0/3012792809 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08ec06ea50 0x7f08ec136e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f093f640 1 -- 172.21.15.100:0/3012792809 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08ec13c9f0 msgr2=0x7f08ec09da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f093f640 1 --2- 172.21.15.100:0/3012792809 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08ec13c9f0 0x7f08ec09da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f093f640 10 monclient: _finish_hunting 0 2024-09-13T11:53:07.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f093f640 1 monclient: found mon.noname-b 2024-09-13T11:53:07.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f093f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:07.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f093f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.122:3300/0 2024-09-13T11:53:07.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f08f093f640 1 -- 172.21.15.100:0/3012792809 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08ec120bb0 con 0x7f08ec06ee30 2024-09-13T11:53:07.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f08f1140640 1 --2- 172.21.15.100:0/3012792809 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f08ec06ea50 0x7f08ec136e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:07.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f08f093f640 1 --2- 172.21.15.100:0/3012792809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08ec06ee30 0x7f08ec137370 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00c430 tx=0x7f08dc00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f08f1941640 10 monclient: get_auth_request con 0x7f08ec13c9f0 auth_method 0 2024-09-13T11:53:07.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f08f1941640 1 --2- 172.21.15.100:0/3012792809 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08ec13c9f0 0x7f08ec09da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:53:07.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f9c577fe640 1 --2- 172.21.15.100:0/1088148368 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9c580a2860 0x7f9c58137960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:07.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f08e1ffb640 1 -- 172.21.15.100:0/3012792809 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc017070 con 0x7f08ec06ee30 2024-09-13T11:53:07.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f08e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f08e1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:07.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f08e1ffb640 10 monclient: dump: 2024-09-13T11:53:07.834 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.834 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.834 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.834 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.834 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.834 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.834 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.834 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.834 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.834 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f08e1ffb640 10 monclient: _finish_auth 0 2024-09-13T11:53:07.835 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f08e1ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:53:07.835 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f08e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:07.835 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f08e1ffb640 1 -- 172.21.15.100:0/3012792809 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08dc005ce0 con 0x7f08ec06ee30 2024-09-13T11:53:07.835 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f08e1ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:07.835 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f08e1ffb640 1 -- 172.21.15.100:0/3012792809 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc0072d0 con 0x7f08ec06ee30 2024-09-13T11:53:07.835 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f08e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.835 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f08e1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:53:07.835 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f08e1ffb640 10 monclient: dump: 2024-09-13T11:53:07.835 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.835 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.835 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.835 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.835 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.835 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.836 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.836 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.836 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.836 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.836 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f9c57fff640 1 --2- 172.21.15.100:0/1088148368 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c5806ea50 0x7f9c58137420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9c5000c870 tx=0x7f9c5000cd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f08f33a4640 5 monclient: authenticate success, global_id 4273 2024-09-13T11:53:07.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f08e37fe640 4 set_mon_vals no callback set 2024-09-13T11:53:07.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f08f33a4640 10 monclient: _renew_subs 2024-09-13T11:53:07.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f08f33a4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:53:07.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7fa06dffb640 1 -- 172.21.15.100:0/2425434581 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fa0740127a0 con 0x7fa07809fb90 2024-09-13T11:53:07.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f08f33a4640 1 -- 172.21.15.100:0/3012792809 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08ec120e90 con 0x7f08ec06ee30 2024-09-13T11:53:07.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f70ff7fe640 1 -- 172.21.15.100:0/1524224731 learned_addr learned my addr 172.21.15.100:0/1524224731 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:07.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f7105b64640 10 monclient(hunting): picked mon.noname-c con 0x7f7100132170 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:07.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f7105b64640 1 --2- 172.21.15.100:0/1524224731 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f710013a820 0x7f71001c6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f9c54ff9640 1 -- 172.21.15.100:0/1088148368 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c50003b70 con 0x7f9c5806ea50 2024-09-13T11:53:07.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f9c54ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f9c54ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:07.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f9c54ff9640 10 monclient: dump: 2024-09-13T11:53:07.839 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.839 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.839 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.839 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.839 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.839 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.839 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.839 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.840 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.840 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f9c54ff9640 10 monclient: _finish_auth 0 2024-09-13T11:53:07.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f9c54ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:53:07.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f9c54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:07.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f08f33a4640 10 monclient: _renew_subs 2024-09-13T11:53:07.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f9c54ff9640 1 -- 172.21.15.100:0/1088148368 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c50003d10 con 0x7f9c5806ea50 2024-09-13T11:53:07.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f08f33a4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:53:07.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f7105b64640 10 monclient(hunting): picked mon.noname-b con 0x7f710013a820 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:07.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f9c5e356640 5 monclient: authenticate success, global_id 4276 2024-09-13T11:53:07.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f9c5e356640 10 monclient: _renew_subs 2024-09-13T11:53:07.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f08f33a4640 1 -- 172.21.15.100:0/3012792809 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08ec1208d0 con 0x7f08ec06ee30 2024-09-13T11:53:07.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f7105b64640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f7105b64640 1 -- 172.21.15.100:0/1524224731 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f710009d810 con 0x7f7100131d90 2024-09-13T11:53:07.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f7105b64640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f7105b64640 1 -- 172.21.15.100:0/1524224731 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f710009d690 con 0x7f710013a820 2024-09-13T11:53:07.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f7105b64640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:07.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f7105b64640 1 -- 172.21.15.100:0/1524224731 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f710009d990 con 0x7f7100132170 2024-09-13T11:53:07.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f7105b64640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:07.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f7105b64640 10 monclient(hunting): authenticate will time out at 768.285094s 2024-09-13T11:53:07.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70ff7fe640 10 monclient(hunting): get_auth_request con 0x7f7100131d90 auth_method 0 2024-09-13T11:53:07.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f9c5e356640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:53:07.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70fffff640 1 --2- 172.21.15.100:0/1524224731 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f710013a820 0x7f71001c6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70fffff640 10 monclient(hunting): get_auth_request con 0x7f710013a820 auth_method 0 2024-09-13T11:53:07.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70fffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:07.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:07.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:07.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.691+0000 7f70feffd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7100132170 0x7f71000745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f9c5e356640 1 -- 172.21.15.100:0/1088148368 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c58120c40 con 0x7f9c5806ea50 2024-09-13T11:53:07.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70feffd640 10 monclient(hunting): get_auth_request con 0x7f7100132170 auth_method 0 2024-09-13T11:53:07.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:07.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70feffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:07.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:07.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70ff7fe640 10 monclient(hunting): handle_auth_done global_id 4290 payload 824 2024-09-13T11:53:07.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70ff7fe640 1 -- 172.21.15.100:0/1524224731 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7100132170 msgr2=0x7f71000745a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70ff7fe640 1 --2- 172.21.15.100:0/1524224731 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7100132170 0x7f71000745a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70ff7fe640 1 -- 172.21.15.100:0/1524224731 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f710013a820 msgr2=0x7f71001c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70ff7fe640 1 --2- 172.21.15.100:0/1524224731 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f710013a820 0x7f71001c6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70ff7fe640 10 monclient: _finish_hunting 0 2024-09-13T11:53:07.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70ff7fe640 1 monclient: found mon.noname-a 2024-09-13T11:53:07.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:07.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70ff7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:07.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70ff7fe640 1 -- 172.21.15.100:0/1524224731 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71000701a0 con 0x7f7100131d90 2024-09-13T11:53:07.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70feffd640 1 --2- 172.21.15.100:0/1524224731 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7100132170 0x7f71000745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:53:07.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f9c54ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:07.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f9c54ff9640 1 -- 172.21.15.100:0/1088148368 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c5000aa70 con 0x7f9c5806ea50 2024-09-13T11:53:07.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f9c54ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7f70ff7fe640 1 --2- 172.21.15.100:0/1524224731 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7100131d90 0x7f71001c5d60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f70f0038040 tx=0x7f70f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7fa06dffb640 1 --2- 172.21.15.100:0/2425434581 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fa05c03dfd0 0x7fa05c040490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7fa07f5f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:07.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.692+0000 7fa07f5f3640 1 -- 172.21.15.100:0/2425434581 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa078073650 con 0x7fa07809fb90 2024-09-13T11:53:07.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7fa06dffb640 1 -- 172.21.15.100:0/2425434581 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7fa074052ba0 con 0x7fa07809fb90 2024-09-13T11:53:07.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7fa06dffb640 1 --2- 172.21.15.100:0/2425434581 >> [v2:172.21.15.100:6800/2354314235,v1:172.21.15.100:6801/2354314235] conn(0x7fa05c043e40 0x7fa05c046280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7fa06dffb640 1 -- 172.21.15.100:0/2425434581 --> [v2:172.21.15.100:6800/2354314235,v1:172.21.15.100:6801/2354314235] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa074014e50 con 0x7fa05c043e40 2024-09-13T11:53:07.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f9c54ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:53:07.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f9c54ff9640 10 monclient: dump: 2024-09-13T11:53:07.848 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.848 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.848 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.848 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.848 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.848 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.848 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.848 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.848 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.848 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f9c5e356640 10 monclient: _renew_subs 2024-09-13T11:53:07.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f9c5e356640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:53:07.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f9c5e356640 1 -- 172.21.15.100:0/1088148368 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c580a0220 con 0x7f9c5806ea50 2024-09-13T11:53:07.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f70e7fff640 1 -- 172.21.15.100:0/1524224731 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f003a050 con 0x7f7100131d90 2024-09-13T11:53:07.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f70e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f9c567fc640 4 set_mon_vals no callback set 2024-09-13T11:53:07.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f70e7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:07.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f70e7fff640 10 monclient: dump: 2024-09-13T11:53:07.851 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.851 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.851 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.851 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.851 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.851 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.851 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.852 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.852 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.852 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f70e7fff640 10 monclient: _finish_auth 0 2024-09-13T11:53:07.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f70e7fff640 10 monclient: _check_auth_tickets 2024-09-13T11:53:07.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f70e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:07.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f70e7fff640 1 -- 172.21.15.100:0/1524224731 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70f0046070 con 0x7f7100131d90 2024-09-13T11:53:07.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f7105b64640 5 monclient: authenticate success, global_id 4290 2024-09-13T11:53:07.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f7105b64640 10 monclient: _renew_subs 2024-09-13T11:53:07.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f7105b64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:07.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f7105b64640 1 -- 172.21.15.100:0/1524224731 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f710009d380 con 0x7f7100131d90 2024-09-13T11:53:07.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f7105b64640 10 monclient: _renew_subs 2024-09-13T11:53:07.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f7105b64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:07.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f7105b64640 1 -- 172.21.15.100:0/1524224731 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7100072060 con 0x7f7100131d90 2024-09-13T11:53:07.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f70e7fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:07.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f70e7fff640 1 -- 172.21.15.100:0/1524224731 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f0035c60 con 0x7f7100131d90 2024-09-13T11:53:07.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f70e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:07.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.694+0000 7f70e7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:07.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.694+0000 7f70e7fff640 10 monclient: dump: 2024-09-13T11:53:07.853 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:07.853 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:07.853 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.853 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:07.853 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:07.853 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:07.853 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:07.853 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:07.854 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:07.854 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:07.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.694+0000 7f9c5e356640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:53:07.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.694+0000 7f9c5e356640 1 -- 172.21.15.100:0/1088148368 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9c5806ee30 con 0x7f9c5806ea50 2024-09-13T11:53:07.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.694+0000 7f9c54ff9640 1 -- 172.21.15.100:0/1088148368 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f9c5000ac10 con 0x7f9c5806ea50 2024-09-13T11:53:07.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7fa07cb8e640 1 --2- 172.21.15.100:0/2425434581 >> [v2:172.21.15.100:6800/2354314235,v1:172.21.15.100:6801/2354314235] conn(0x7fa05c043e40 0x7fa05c046280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7fa07cb8e640 10 monclient: get_auth_request con 0x7fa05c043e40 auth_method 0 2024-09-13T11:53:07.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.693+0000 7f70fdffb640 4 set_mon_vals no callback set 2024-09-13T11:53:07.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.694+0000 7fa07d38f640 1 --2- 172.21.15.100:0/2425434581 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fa05c03dfd0 0x7fa05c040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.694+0000 7fa07d38f640 10 monclient: get_auth_request con 0x7fa05c03dfd0 auth_method 0 2024-09-13T11:53:07.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.694+0000 7f08e1ffb640 1 -- 172.21.15.100:0/3012792809 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f08dc01a070 con 0x7f08ec06ee30 2024-09-13T11:53:07.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.694+0000 7f08e1ffb640 1 --2- 172.21.15.100:0/3012792809 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f08d003dfa0 0x7f08d0040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.694+0000 7f08e1ffb640 1 -- 172.21.15.100:0/3012792809 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7f08dc0519c0 con 0x7f08ec06ee30 2024-09-13T11:53:07.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.694+0000 7f08f33a4640 1 --2- 172.21.15.100:0/3012792809 >> [v2:172.21.15.122:6800/904725860,v1:172.21.15.122:6801/904725860] conn(0x7f08ec09e480 0x7f08ec09e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.694+0000 7fa06dffb640 1 -- 172.21.15.100:0/2425434581 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7fa074012a90 con 0x7fa07809fb90 2024-09-13T11:53:07.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.694+0000 7fa06dffb640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-13T11:53:07.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.694+0000 7f08f33a4640 1 -- 172.21.15.100:0/3012792809 --> [v2:172.21.15.122:6800/904725860,v1:172.21.15.122:6801/904725860] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f08ec0a2860 con 0x7f08ec09e480 2024-09-13T11:53:07.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.694+0000 7fa07cb8e640 1 --2- 172.21.15.100:0/2425434581 >> [v2:172.21.15.100:6800/2354314235,v1:172.21.15.100:6801/2354314235] conn(0x7fa05c043e40 0x7fa05c046280 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.694+0000 7f08f1140640 1 --2- 172.21.15.100:0/3012792809 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f08d003dfa0 0x7f08d0040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.694+0000 7f9c54ff9640 1 --2- 172.21.15.100:0/1088148368 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9c4403dff0 0x7f9c440404b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.694+0000 7f08f1140640 10 monclient: get_auth_request con 0x7f08d003dfa0 auth_method 0 2024-09-13T11:53:07.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.694+0000 7f9c54ff9640 1 -- 172.21.15.100:0/1088148368 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7f9c500524f0 con 0x7f9c5806ea50 2024-09-13T11:53:07.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.694+0000 7f9c54ff9640 1 --2- 172.21.15.100:0/1088148368 >> [v2:172.21.15.122:6808/3206511700,v1:172.21.15.122:6809/3206511700] conn(0x7f9c44043e60 0x7f9c440462a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.695+0000 7f7105b64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:07.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.695+0000 7f7105b64640 1 -- 172.21.15.100:0/1524224731 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f710009dec0 con 0x7f7100131d90 2024-09-13T11:53:07.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.695+0000 7fa07d38f640 1 --2- 172.21.15.100:0/2425434581 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fa05c03dfd0 0x7fa05c040490 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa070034000 tx=0x7fa07002e000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.694+0000 7f9c54ff9640 1 -- 172.21.15.100:0/1088148368 --> [v2:172.21.15.122:6808/3206511700,v1:172.21.15.122:6809/3206511700] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9c50012e50 con 0x7f9c44043e60 2024-09-13T11:53:07.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.694+0000 7f9c54ff9640 1 -- 172.21.15.100:0/1088148368 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f9c50059050 con 0x7f9c5806ea50 2024-09-13T11:53:07.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.694+0000 7f9c54ff9640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-13T11:53:07.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.694+0000 7f9c577fe640 1 --2- 172.21.15.100:0/1088148368 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9c4403dff0 0x7f9c440404b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.695+0000 7f9c577fe640 10 monclient: get_auth_request con 0x7f9c4403dff0 auth_method 0 2024-09-13T11:53:07.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.695+0000 7f08f1941640 1 --2- 172.21.15.100:0/3012792809 >> [v2:172.21.15.122:6800/904725860,v1:172.21.15.122:6801/904725860] conn(0x7f08ec09e480 0x7f08ec09e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.695+0000 7f70e7fff640 1 -- 172.21.15.100:0/1524224731 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f70f004b460 con 0x7f7100131d90 2024-09-13T11:53:07.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.695+0000 7f08f1941640 10 monclient: get_auth_request con 0x7f08ec09e480 auth_method 0 2024-09-13T11:53:07.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.695+0000 7f9c5c8f3640 1 --2- 172.21.15.100:0/1088148368 >> [v2:172.21.15.122:6808/3206511700,v1:172.21.15.122:6809/3206511700] conn(0x7f9c44043e60 0x7f9c440462a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.695+0000 7f9c5c8f3640 10 monclient: get_auth_request con 0x7f9c44043e60 auth_method 0 2024-09-13T11:53:07.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.695+0000 7f9c577fe640 1 --2- 172.21.15.100:0/1088148368 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9c4403dff0 0x7f9c440404b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9c58120b70 tx=0x7f9c4c000f10 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.695+0000 7f9c5c8f3640 1 --2- 172.21.15.100:0/1088148368 >> [v2:172.21.15.122:6808/3206511700,v1:172.21.15.122:6809/3206511700] conn(0x7f9c44043e60 0x7f9c440462a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.695+0000 7fa06dffb640 1 -- 172.21.15.100:0/2425434581 <== osd.2 v2:172.21.15.100:6800/2354314235 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa074014e50 con 0x7fa05c043e40 2024-09-13T11:53:07.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.695+0000 7f70e7fff640 1 --2- 172.21.15.100:0/1524224731 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f70ec03e020 0x7f70ec0404e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.695+0000 7f70e7fff640 1 -- 172.21.15.100:0/1524224731 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7f70f0077a90 con 0x7f7100131d90 2024-09-13T11:53:07.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.695+0000 7f70e7fff640 1 --2- 172.21.15.100:0/1524224731 >> [v2:172.21.15.100:6816/92840425,v1:172.21.15.100:6817/92840425] conn(0x7f70ec043f10 0x7f70ec046350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:07.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.695+0000 7f08f1941640 1 --2- 172.21.15.100:0/3012792809 >> [v2:172.21.15.122:6800/904725860,v1:172.21.15.122:6801/904725860] conn(0x7f08ec09e480 0x7f08ec09e860 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.695+0000 7f70e7fff640 1 -- 172.21.15.100:0/1524224731 --> [v2:172.21.15.100:6816/92840425,v1:172.21.15.100:6817/92840425] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f70f000b650 con 0x7f70ec043f10 2024-09-13T11:53:07.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.696+0000 7f70fffff640 1 --2- 172.21.15.100:0/1524224731 >> [v2:172.21.15.100:6816/92840425,v1:172.21.15.100:6817/92840425] conn(0x7f70ec043f10 0x7f70ec046350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.696+0000 7f70fffff640 10 monclient: get_auth_request con 0x7f70ec043f10 auth_method 0 2024-09-13T11:53:07.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.696+0000 7f9c54ff9640 1 -- 172.21.15.100:0/1088148368 <== osd.3 v2:172.21.15.122:6808/3206511700 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9c50012e50 con 0x7f9c44043e60 2024-09-13T11:53:07.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.696+0000 7f70e7fff640 1 -- 172.21.15.100:0/1524224731 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f70f007c220 con 0x7f7100131d90 2024-09-13T11:53:07.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.696+0000 7f70fffff640 1 --2- 172.21.15.100:0/1524224731 >> [v2:172.21.15.100:6816/92840425,v1:172.21.15.100:6817/92840425] conn(0x7f70ec043f10 0x7f70ec046350 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.696+0000 7f70e7fff640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-13T11:53:07.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.696+0000 7f08e1ffb640 1 -- 172.21.15.100:0/3012792809 <== osd.5 v2:172.21.15.122:6800/904725860 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f08ec0a2860 con 0x7f08ec09e480 2024-09-13T11:53:07.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.697+0000 7f70e7fff640 1 -- 172.21.15.100:0/1524224731 <== osd.0 v2:172.21.15.100:6816/92840425 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f70f000b650 con 0x7f70ec043f10 2024-09-13T11:53:07.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.706+0000 7f70feffd640 1 --2- 172.21.15.100:0/1524224731 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f70ec03e020 0x7f70ec0404e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:07.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.706+0000 7f70feffd640 10 monclient: get_auth_request con 0x7f70ec03e020 auth_method 0 2024-09-13T11:53:07.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.707+0000 7f08f1140640 1 --2- 172.21.15.100:0/3012792809 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f08d003dfa0 0x7f08d0040460 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f08e8000c00 tx=0x7f08e802e000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.707+0000 7f70feffd640 1 --2- 172.21.15.100:0/1524224731 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f70ec03e020 0x7f70ec0404e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f70f4007fd0 tx=0x7f70f4009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:07.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.709+0000 7fa07f5f3640 1 -- 172.21.15.100:0/2425434581 --> [v2:172.21.15.100:6800/2354314235,v1:172.21.15.100:6801/2354314235] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa07812ca60 con 0x7fa05c043e40 2024-09-13T11:53:07.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.709+0000 7fa06dffb640 1 -- 172.21.15.100:0/2425434581 <== osd.2 v2:172.21.15.100:6800/2354314235 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa07812ca60 con 0x7fa05c043e40 2024-09-13T11:53:07.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.710+0000 7f08f33a4640 1 -- 172.21.15.100:0/3012792809 --> [v2:172.21.15.122:6800/904725860,v1:172.21.15.122:6801/904725860] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f08ec0a0190 con 0x7f08ec09e480 2024-09-13T11:53:07.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.710+0000 7fa07f5f3640 1 -- 172.21.15.100:0/2425434581 >> [v2:172.21.15.100:6800/2354314235,v1:172.21.15.100:6801/2354314235] conn(0x7fa05c043e40 msgr2=0x7fa05c046280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.710+0000 7fa07f5f3640 1 --2- 172.21.15.100:0/2425434581 >> [v2:172.21.15.100:6800/2354314235,v1:172.21.15.100:6801/2354314235] conn(0x7fa05c043e40 0x7fa05c046280 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.710+0000 7fa07f5f3640 1 -- 172.21.15.100:0/2425434581 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fa05c03dfd0 msgr2=0x7fa05c040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.710+0000 7fa07f5f3640 1 --2- 172.21.15.100:0/2425434581 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fa05c03dfd0 0x7fa05c040490 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa070034000 tx=0x7fa07002e000 comp rx=0 tx=0).stop 2024-09-13T11:53:07.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.710+0000 7fa07f5f3640 10 monclient: shutdown 2024-09-13T11:53:07.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.710+0000 7fa07f5f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:07.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.710+0000 7fa07f5f3640 1 -- 172.21.15.100:0/2425434581 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa07809fb90 msgr2=0x7fa07813dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.710+0000 7fa07f5f3640 1 --2- 172.21.15.100:0/2425434581 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa07809fb90 0x7fa07813dc50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa07400ca50 tx=0x7fa07400cf10 comp rx=0 tx=0).stop 2024-09-13T11:53:07.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.710+0000 7f08e1ffb640 1 -- 172.21.15.100:0/3012792809 <== osd.5 v2:172.21.15.122:6800/904725860 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f08ec0a0190 con 0x7f08ec09e480 2024-09-13T11:53:07.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.710+0000 7fa07db90640 1 -- 172.21.15.100:0/2425434581 reap_dead start 2024-09-13T11:53:07.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.710+0000 7fa07f5f3640 1 -- 172.21.15.100:0/2425434581 shutdown_connections 2024-09-13T11:53:07.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.710+0000 7fa07f5f3640 1 -- 172.21.15.100:0/2425434581 >> 172.21.15.100:0/2425434581 conn(0x7fa0780a8660 msgr2=0x7fa078071000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:07.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.710+0000 7fa07f5f3640 1 -- 172.21.15.100:0/2425434581 shutdown_connections 2024-09-13T11:53:07.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.710+0000 7f9c5e356640 1 -- 172.21.15.100:0/1088148368 --> [v2:172.21.15.122:6808/3206511700,v1:172.21.15.122:6809/3206511700] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9c580a0a40 con 0x7f9c44043e60 2024-09-13T11:53:07.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.710+0000 7f08f33a4640 1 -- 172.21.15.100:0/3012792809 >> [v2:172.21.15.122:6800/904725860,v1:172.21.15.122:6801/904725860] conn(0x7f08ec09e480 msgr2=0x7f08ec09e860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.710+0000 7f08f33a4640 1 --2- 172.21.15.100:0/3012792809 >> [v2:172.21.15.122:6800/904725860,v1:172.21.15.122:6801/904725860] conn(0x7f08ec09e480 0x7f08ec09e860 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.710+0000 7fa07f5f3640 1 -- 172.21.15.100:0/2425434581 wait complete. 2024-09-13T11:53:07.872 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f08f33a4640 1 -- 172.21.15.100:0/3012792809 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f08d003dfa0 msgr2=0x7f08d0040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.872 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f08f33a4640 1 --2- 172.21.15.100:0/3012792809 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f08d003dfa0 0x7f08d0040460 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f08e8000c00 tx=0x7f08e802e000 comp rx=0 tx=0).stop 2024-09-13T11:53:07.872 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f08f33a4640 10 monclient: shutdown 2024-09-13T11:53:07.872 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f08f33a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:07.872 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f08f33a4640 1 -- 172.21.15.100:0/3012792809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08ec06ee30 msgr2=0x7f08ec137370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.872 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f08f33a4640 1 --2- 172.21.15.100:0/3012792809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08ec06ee30 0x7f08ec137370 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00c430 tx=0x7f08dc00c8f0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.872 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f08f1941640 1 -- 172.21.15.100:0/3012792809 reap_dead start 2024-09-13T11:53:07.872 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f08f33a4640 1 -- 172.21.15.100:0/3012792809 shutdown_connections 2024-09-13T11:53:07.872 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f08f33a4640 1 -- 172.21.15.100:0/3012792809 >> 172.21.15.100:0/3012792809 conn(0x7f08ec0a68e0 msgr2=0x7f08ec0a8470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:07.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f7105b64640 1 -- 172.21.15.100:0/1524224731 --> [v2:172.21.15.100:6816/92840425,v1:172.21.15.100:6817/92840425] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f71000a3d80 con 0x7f70ec043f10 2024-09-13T11:53:07.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f08f33a4640 1 -- 172.21.15.100:0/3012792809 shutdown_connections 2024-09-13T11:53:07.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f08f33a4640 1 -- 172.21.15.100:0/3012792809 wait complete. 2024-09-13T11:53:07.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f9c54ff9640 1 -- 172.21.15.100:0/1088148368 <== osd.3 v2:172.21.15.122:6808/3206511700 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9c580a0a40 con 0x7f9c44043e60 2024-09-13T11:53:07.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f70e7fff640 1 -- 172.21.15.100:0/1524224731 <== osd.0 v2:172.21.15.100:6816/92840425 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f71000a3d80 con 0x7f70ec043f10 2024-09-13T11:53:07.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f9c5e356640 1 -- 172.21.15.100:0/1088148368 >> [v2:172.21.15.122:6808/3206511700,v1:172.21.15.122:6809/3206511700] conn(0x7f9c44043e60 msgr2=0x7f9c440462a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f9c5e356640 1 --2- 172.21.15.100:0/1088148368 >> [v2:172.21.15.122:6808/3206511700,v1:172.21.15.122:6809/3206511700] conn(0x7f9c44043e60 0x7f9c440462a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f9c5e356640 1 -- 172.21.15.100:0/1088148368 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9c4403dff0 msgr2=0x7f9c440404b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f9c5e356640 1 --2- 172.21.15.100:0/1088148368 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9c4403dff0 0x7f9c440404b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9c58120b70 tx=0x7f9c4c000f10 comp rx=0 tx=0).stop 2024-09-13T11:53:07.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f9c5e356640 10 monclient: shutdown 2024-09-13T11:53:07.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f9c5e356640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:07.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f9c5e356640 1 -- 172.21.15.100:0/1088148368 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c5806ea50 msgr2=0x7f9c58137420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f9c5e356640 1 --2- 172.21.15.100:0/1088148368 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c5806ea50 0x7f9c58137420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9c5000c870 tx=0x7f9c5000cd30 comp rx=0 tx=0).stop 2024-09-13T11:53:07.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f7105b64640 1 -- 172.21.15.100:0/1524224731 >> [v2:172.21.15.100:6816/92840425,v1:172.21.15.100:6817/92840425] conn(0x7f70ec043f10 msgr2=0x7f70ec046350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f9c5c8f3640 1 -- 172.21.15.100:0/1088148368 reap_dead start 2024-09-13T11:53:07.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f7105b64640 1 --2- 172.21.15.100:0/1524224731 >> [v2:172.21.15.100:6816/92840425,v1:172.21.15.100:6817/92840425] conn(0x7f70ec043f10 0x7f70ec046350 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f9c5e356640 1 -- 172.21.15.100:0/1088148368 shutdown_connections 2024-09-13T11:53:07.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.711+0000 7f9c5e356640 1 -- 172.21.15.100:0/1088148368 >> 172.21.15.100:0/1088148368 conn(0x7f9c580a68e0 msgr2=0x7f9c5813e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:07.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.712+0000 7f7105b64640 1 -- 172.21.15.100:0/1524224731 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f70ec03e020 msgr2=0x7f70ec0404e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.712+0000 7f7105b64640 1 --2- 172.21.15.100:0/1524224731 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f70ec03e020 0x7f70ec0404e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f70f4007fd0 tx=0x7f70f4009040 comp rx=0 tx=0).stop 2024-09-13T11:53:07.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.712+0000 7f7105b64640 10 monclient: shutdown 2024-09-13T11:53:07.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.712+0000 7f7105b64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:07.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.712+0000 7f9c5e356640 1 -- 172.21.15.100:0/1088148368 shutdown_connections 2024-09-13T11:53:07.878 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.712+0000 7f7105b64640 1 -- 172.21.15.100:0/1524224731 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7100131d90 msgr2=0x7f71001c5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:07.878 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.712+0000 7f7105b64640 1 --2- 172.21.15.100:0/1524224731 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7100131d90 0x7f71001c5d60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f70f0038040 tx=0x7f70f00046c0 comp rx=0 tx=0).stop 2024-09-13T11:53:07.878 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.712+0000 7f9c5e356640 1 -- 172.21.15.100:0/1088148368 wait complete. 2024-09-13T11:53:07.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.712+0000 7f70fffff640 1 -- 172.21.15.100:0/1524224731 reap_dead start 2024-09-13T11:53:07.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.712+0000 7f7105b64640 1 -- 172.21.15.100:0/1524224731 shutdown_connections 2024-09-13T11:53:07.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.712+0000 7f7105b64640 1 -- 172.21.15.100:0/1524224731 >> 172.21.15.100:0/1524224731 conn(0x7f71000a2c90 msgr2=0x7f7100139850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:07.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.712+0000 7f7105b64640 1 -- 172.21.15.100:0/1524224731 shutdown_connections 2024-09-13T11:53:07.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:07.712+0000 7f7105b64640 1 -- 172.21.15.100:0/1524224731 wait complete. 2024-09-13T11:53:07.880 INFO:teuthology.orchestra.run.smithi100.stdout:47244640261 2024-09-13T11:53:07.880 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T11:53:07.881 INFO:teuthology.orchestra.run.smithi100.stdout:47244640262 2024-09-13T11:53:07.881 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T11:53:07.882 INFO:teuthology.orchestra.run.smithi100.stdout:51539607558 2024-09-13T11:53:07.882 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T11:53:07.883 INFO:teuthology.orchestra.run.smithi100.stdout:51539607557 2024-09-13T11:53:07.883 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T11:53:08.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.033+0000 7f57e06b6640 10 monclient: get_monmap_and_config 2024-09-13T11:53:08.036 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.033+0000 7f9659649640 10 monclient: get_monmap_and_config 2024-09-13T11:53:08.036 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.033+0000 7f9659649640 10 monclient: build_initial_monmap 2024-09-13T11:53:08.036 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.033+0000 7f9659649640 10 monclient: monmap: 2024-09-13T11:53:08.036 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:08.036 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.036 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:08.034368+0000 2024-09-13T11:53:08.037 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:08.034368+0000 2024-09-13T11:53:08.037 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:08.037 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.038 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:08.038 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:08.038 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:08.038 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:08.038 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:08.038 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.033+0000 7f57e06b6640 10 monclient: build_initial_monmap 2024-09-13T11:53:08.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.033+0000 7f57e06b6640 10 monclient: monmap: 2024-09-13T11:53:08.040 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:08.040 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.041 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:08.034267+0000 2024-09-13T11:53:08.041 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:08.034267+0000 2024-09-13T11:53:08.041 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:08.041 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.041 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:08.041 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:08.042 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:08.042 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:08.042 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:08.042 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f57e06b6640 1 Processor -- start 2024-09-13T11:53:08.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f57e06b6640 1 -- start start 2024-09-13T11:53:08.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f57e06b6640 10 monclient: init 2024-09-13T11:53:08.044 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f9659649640 1 Processor -- start 2024-09-13T11:53:08.044 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f9659649640 1 -- start start 2024-09-13T11:53:08.044 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f9659649640 10 monclient: init 2024-09-13T11:53:08.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f9659649640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:08.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f9659649640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-13T11:53:08.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f57e06b6640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:08.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f9659649640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f965406ea50 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:08.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f9659649640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f96540a2860 0x7f965406f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f9659649640 10 monclient(hunting): picked mon.noname-c con 0x7f96540a2860 addr v2:172.21.15.122:3300/0 2024-09-13T11:53:08.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f9659649640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f965413c9f0 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:08.048 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f9659649640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f965406fa80 con 0x7f965413c9f0 2024-09-13T11:53:08.048 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f9659649640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96540a1590 con 0x7f965406ea50 2024-09-13T11:53:08.048 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f9659649640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f57e06b6640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-13T11:53:08.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f57e06b6640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f57d806ea50 0x7f57d80a2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f57e06b6640 10 monclient(hunting): picked mon.noname-a con 0x7f57d806ea50 addr v2:172.21.15.100:3300/0 2024-09-13T11:53:08.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f57e06b6640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57d806ee30 0x7f57d806f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f9659649640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f96540a2cb0 con 0x7f96540a2860 2024-09-13T11:53:08.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f9659649640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:08.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f9659649640 10 monclient(hunting): authenticate will time out at 768.627936s 2024-09-13T11:53:08.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f57e06b6640 10 monclient(hunting): picked mon.noname-b con 0x7f57d806ee30 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:08.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f57e06b6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f57d806f790 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:08.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f57e06b6640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57d80a1590 con 0x7f57d806f790 2024-09-13T11:53:08.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f57e06b6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f96537fe640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f965413c9f0 0x7f965413ede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:57968/0 (socket says 172.21.15.100:57968) 2024-09-13T11:53:08.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f96537fe640 1 -- 172.21.15.100:0/4176077449 learned_addr learned my addr 172.21.15.100:0/4176077449 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:53:08.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f57e06b6640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f57d80a1870 con 0x7f57d806ea50 2024-09-13T11:53:08.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f964bfff640 1 --2- 172.21.15.100:0/4176077449 >> v2:172.21.15.122:3300/0 conn(0x7f96540a2860 0x7f965406f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.055 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f57e06b6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.055 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f57e06b6640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f57d80a1b50 con 0x7f57d806ee30 2024-09-13T11:53:08.055 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.034+0000 7f57e06b6640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:08.056 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f964bfff640 10 monclient(hunting): get_auth_request con 0x7f96540a2860 auth_method 0 2024-09-13T11:53:08.057 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f57de452640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f57d806ea50 0x7f57d80a2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.057 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f57de452640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f57d806ea50 0x7f57d80a2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49524/0 (socket says 172.21.15.100:49524) 2024-09-13T11:53:08.057 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f57de452640 1 -- 172.21.15.100:0/1277660451 learned_addr learned my addr 172.21.15.100:0/1277660451 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:08.057 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f57e06b6640 10 monclient(hunting): authenticate will time out at 768.628566s 2024-09-13T11:53:08.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f57ddc51640 1 --2- 172.21.15.100:0/1277660451 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57d806ee30 0x7f57d806f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f57de452640 10 monclient(hunting): get_auth_request con 0x7f57d806ea50 auth_method 0 2024-09-13T11:53:08.059 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f964bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.059 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f964bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.059 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f964bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.059 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f964bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:08.059 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f964bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:08.060 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f964bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:08.062 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f57de452640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.062 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f57de452640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.062 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f57de452640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.063 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f57ddc51640 10 monclient(hunting): get_auth_request con 0x7f57d806ee30 auth_method 0 2024-09-13T11:53:08.063 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f57ddc51640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.063 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f57ddc51640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.063 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f57ddc51640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.064 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f57de452640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:08.064 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f57de452640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:08.064 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f57de452640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:08.064 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.036+0000 7f57de452640 10 monclient(hunting): handle_auth_done global_id 4296 payload 274 2024-09-13T11:53:08.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f964bfff640 10 monclient(hunting): handle_auth_done global_id 4282 payload 274 2024-09-13T11:53:08.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f964bfff640 1 -- 172.21.15.100:0/4176077449 >> v1:172.21.15.122:6789/0 conn(0x7f965406ea50 legacy=0x7f965406ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:08.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f964bfff640 1 -- 172.21.15.100:0/4176077449 >> v1:172.21.15.100:6789/0 conn(0x7f965413c9f0 legacy=0x7f965413ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f964bfff640 10 monclient: _finish_hunting 0 2024-09-13T11:53:08.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f964bfff640 1 monclient: found mon.noname-c 2024-09-13T11:53:08.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f964bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:08.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f964bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.122:3300/0 2024-09-13T11:53:08.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.035+0000 7f964bfff640 1 -- 172.21.15.100:0/4176077449 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96540a1870 con 0x7f96540a2860 2024-09-13T11:53:08.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.036+0000 7f57dd450640 1 -- 172.21.15.100:0/1277660451 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3873444653 0 0) 0x7f57d80a1590 con 0x7f57d806f790 2024-09-13T11:53:08.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.036+0000 7f57de452640 1 -- 172.21.15.100:0/1277660451 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57d806ee30 msgr2=0x7f57d806f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:08.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.036+0000 7f57de452640 1 --2- 172.21.15.100:0/1277660451 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57d806ee30 0x7f57d806f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.036+0000 7f57de452640 1 -- 172.21.15.100:0/1277660451 >> v1:172.21.15.100:6789/0 conn(0x7f57d806f790 legacy=0x7f57d81375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.036+0000 7f57de452640 10 monclient: _finish_hunting 0 2024-09-13T11:53:08.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.036+0000 7f57de452640 1 monclient: found mon.noname-a 2024-09-13T11:53:08.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.036+0000 7f57de452640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:08.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.036+0000 7f57de452640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:08.072 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.036+0000 7f57de452640 1 -- 172.21.15.100:0/1277660451 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57d806fca0 con 0x7f57d806ea50 2024-09-13T11:53:08.072 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.036+0000 7f57dd450640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:53:08.072 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.036+0000 7f57ddc51640 1 --2- 172.21.15.100:0/1277660451 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57d806ee30 0x7f57d806f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:53:08.072 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f57de452640 1 --2- 172.21.15.100:0/1277660451 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f57d806ea50 0x7f57d80a2860 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f57d4009870 tx=0x7f57d402a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=549e75858bf2a67c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:08.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f964bfff640 1 --2- 172.21.15.100:0/4176077449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96540a2860 0x7f965406f540 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f963c0097a0 tx=0x7f963c02a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1e935dac3fc6c95 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:08.074 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f57dd450640 1 -- 172.21.15.100:0/1277660451 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d4037070 con 0x7f57d806ea50 2024-09-13T11:53:08.074 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f57dd450640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:08.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f96527fc640 1 -- 172.21.15.100:0/4176077449 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f963c037070 con 0x7f96540a2860 2024-09-13T11:53:08.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f96527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:08.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f96527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:08.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f57dd450640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:08.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f96527fc640 10 monclient: dump: 2024-09-13T11:53:08.076 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:08.077 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.077 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.077 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.077 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:08.077 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.077 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:08.078 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:08.078 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:08.078 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f96527fc640 10 monclient: _finish_auth 0 2024-09-13T11:53:08.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f96527fc640 10 monclient: _check_auth_tickets 2024-09-13T11:53:08.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f96527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:08.079 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f96527fc640 1 -- 172.21.15.100:0/4176077449 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f963c02adb0 con 0x7f96540a2860 2024-09-13T11:53:08.079 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f96527fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:08.079 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f96527fc640 1 -- 172.21.15.100:0/4176077449 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f963c032660 con 0x7f96540a2860 2024-09-13T11:53:08.079 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f96527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:08.079 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f96527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:53:08.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f57dd450640 10 monclient: dump: 2024-09-13T11:53:08.080 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:08.080 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.081 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.081 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.081 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:08.081 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.081 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:08.081 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:08.082 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:08.082 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f57dd450640 10 monclient: _finish_auth 0 2024-09-13T11:53:08.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f57dd450640 10 monclient: _check_auth_tickets 2024-09-13T11:53:08.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f57dd450640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:08.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f57e06b6640 5 monclient: authenticate success, global_id 4296 2024-09-13T11:53:08.083 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f96527fc640 10 monclient: dump: 2024-09-13T11:53:08.083 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:08.083 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.084 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.084 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.084 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:08.084 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.084 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:08.085 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:08.085 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:08.085 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.086 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f57e06b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:08.086 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f57dd450640 1 -- 172.21.15.100:0/1277660451 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57d4032440 con 0x7f57d806ea50 2024-09-13T11:53:08.086 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f57dd450640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:08.086 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f57e06b6640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:08.087 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f57e06b6640 4 set_mon_vals no callback set 2024-09-13T11:53:08.087 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f57dd450640 1 -- 172.21.15.100:0/1277660451 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d4032810 con 0x7f57d806ea50 2024-09-13T11:53:08.087 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f57dd450640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:08.087 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f57dd450640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:08.087 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f57dd450640 10 monclient: dump: 2024-09-13T11:53:08.087 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:08.088 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.088 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.088 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.088 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:08.088 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.089 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:08.089 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:08.089 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:08.089 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.089 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.038+0000 7f57e06b6640 10 monclient: shutdown 2024-09-13T11:53:08.089 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.038+0000 7f57e06b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:08.090 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.038+0000 7f57e06b6640 1 -- 172.21.15.100:0/1277660451 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f57d806ea50 msgr2=0x7f57d80a2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.090 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.038+0000 7f57e06b6640 1 --2- 172.21.15.100:0/1277660451 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f57d806ea50 0x7f57d80a2860 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f57d4009870 tx=0x7f57d402a130 comp rx=0 tx=0).stop 2024-09-13T11:53:08.091 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f9659649640 5 monclient: authenticate success, global_id 4282 2024-09-13T11:53:08.091 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f9659649640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:08.091 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.037+0000 7f9659649640 4 set_mon_vals no callback set 2024-09-13T11:53:08.091 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.038+0000 7f9659649640 10 monclient: shutdown 2024-09-13T11:53:08.091 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.038+0000 7f9659649640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:08.091 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.038+0000 7f9659649640 1 -- 172.21.15.100:0/4176077449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96540a2860 msgr2=0x7f965406f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.091 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.038+0000 7f9659649640 1 --2- 172.21.15.100:0/4176077449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96540a2860 0x7f965406f540 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f963c0097a0 tx=0x7f963c02a060 comp rx=0 tx=0).stop 2024-09-13T11:53:08.091 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.038+0000 7f57e06b6640 1 -- 172.21.15.100:0/1277660451 shutdown_connections 2024-09-13T11:53:08.092 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.038+0000 7f9659649640 1 -- 172.21.15.100:0/4176077449 shutdown_connections 2024-09-13T11:53:08.092 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.038+0000 7f9659649640 1 --2- 172.21.15.100:0/4176077449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96540a2860 0x7f965406f540 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.092 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.038+0000 7f9659649640 1 -- 172.21.15.100:0/4176077449 >> 172.21.15.100:0/4176077449 conn(0x7f96540a68e0 msgr2=0x7f96540a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:08.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.038+0000 7f57e06b6640 1 --2- 172.21.15.100:0/1277660451 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57d806ee30 0x7f57d806f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.038+0000 7f57e06b6640 1 --2- 172.21.15.100:0/1277660451 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f57d806ea50 0x7f57d80a2860 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.038+0000 7f57e06b6640 1 -- 172.21.15.100:0/1277660451 >> 172.21.15.100:0/1277660451 conn(0x7f57d80a68e0 msgr2=0x7f57d80a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:08.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.038+0000 7f9659649640 1 -- 172.21.15.100:0/4176077449 shutdown_connections 2024-09-13T11:53:08.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.038+0000 7f9659649640 1 -- 172.21.15.100:0/4176077449 wait complete. 2024-09-13T11:53:08.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.038+0000 7f57e06b6640 1 -- 172.21.15.100:0/1277660451 shutdown_connections 2024-09-13T11:53:08.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.038+0000 7f57e06b6640 1 -- 172.21.15.100:0/1277660451 wait complete. 2024-09-13T11:53:08.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.039+0000 7f9659649640 10 monclient: build_initial_monmap 2024-09-13T11:53:08.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.039+0000 7f9659649640 10 monclient: monmap: 2024-09-13T11:53:08.094 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:08.094 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:08.095 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:08.095 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:08.095 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:08.095 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.095 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:08.095 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:08.095 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:08.095 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.039+0000 7f57e06b6640 10 monclient: build_initial_monmap 2024-09-13T11:53:08.096 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.039+0000 7f57e06b6640 10 monclient: monmap: 2024-09-13T11:53:08.096 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:08.096 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:08.096 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:08.096 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:08.096 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:08.096 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.096 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:08.096 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:08.096 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:08.096 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.039+0000 7f9659649640 1 Processor -- start 2024-09-13T11:53:08.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.039+0000 7f9659649640 1 -- start start 2024-09-13T11:53:08.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.039+0000 7f9659649640 10 monclient: init 2024-09-13T11:53:08.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.039+0000 7f57e06b6640 1 Processor -- start 2024-09-13T11:53:08.098 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.039+0000 7f57e06b6640 1 -- start start 2024-09-13T11:53:08.098 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.039+0000 7f57e06b6640 10 monclient: init 2024-09-13T11:53:08.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.039+0000 7f9659649640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:08.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9659649640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:53:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9659649640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f965406ea50 0x7f96541cc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9659649640 10 monclient(hunting): picked mon.noname-c con 0x7f965406ea50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9659649640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96540a2860 0x7f9654137880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9659649640 10 monclient(hunting): picked mon.noname-a con 0x7f96540a2860 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9659649640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f965413c9f0 0x7f965409dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9659649640 10 monclient(hunting): picked mon.noname-b con 0x7f965413c9f0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9659649640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9659649640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f96540a5be0 con 0x7f96540a2860 2024-09-13T11:53:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9659649640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9659649640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f96540a5a60 con 0x7f965413c9f0 2024-09-13T11:53:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9659649640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9659649640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f9654140500 con 0x7f965406ea50 2024-09-13T11:53:08.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9659649640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:08.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9659649640 10 monclient(hunting): authenticate will time out at 768.633078s 2024-09-13T11:53:08.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9652ffd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f965406ea50 0x7f96541cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9652ffd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f965406ea50 0x7f96541cc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47354/0 (socket says 172.21.15.100:47354) 2024-09-13T11:53:08.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9652ffd640 1 -- 172.21.15.100:0/1271138643 learned_addr learned my addr 172.21.15.100:0/1271138643 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:08.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9652ffd640 10 monclient(hunting): get_auth_request con 0x7f965406ea50 auth_method 0 2024-09-13T11:53:08.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9652ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9652ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9652ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57e06b6640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57e06b6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:53:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57e06b6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f57d806ea50 0x7f57d8123d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57e06b6640 10 monclient(hunting): picked mon.noname-b con 0x7f57d806ea50 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57e06b6640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f57d806ee30 0x7f57d8124280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57e06b6640 10 monclient(hunting): picked mon.noname-a con 0x7f57d806ee30 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57e06b6640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57d806f790 0x7f57d81247c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57e06b6640 10 monclient(hunting): picked mon.noname-c con 0x7f57d806f790 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57e06b6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57e06b6640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f57d8137e30 con 0x7f57d806ee30 2024-09-13T11:53:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57e06b6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57e06b6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f57d8137cb0 con 0x7f57d806ea50 2024-09-13T11:53:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57e06b6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57e06b6640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f57d8137fb0 con 0x7f57d806f790 2024-09-13T11:53:08.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57e06b6640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:08.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57e06b6640 10 monclient(hunting): authenticate will time out at 768.633469s 2024-09-13T11:53:08.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9652ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:08.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9652ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:08.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9652ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:08.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f964bfff640 1 --2- 172.21.15.100:0/1271138643 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96540a2860 0x7f9654137880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f964bfff640 10 monclient(hunting): get_auth_request con 0x7f96540a2860 auth_method 0 2024-09-13T11:53:08.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f964bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f964bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f964bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f964bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:08.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f964bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:08.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f964bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:08.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57de452640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f57d806ea50 0x7f57d8123d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57ddc51640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f57d806ee30 0x7f57d8124280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57de452640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f57d806ea50 0x7f57d8123d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.100:50440/0 (socket says 172.21.15.100:50440) 2024-09-13T11:53:08.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57ddc51640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f57d806ee30 0x7f57d8124280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49542/0 (socket says 172.21.15.100:49542) 2024-09-13T11:53:08.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57de452640 1 -- 172.21.15.100:0/477806134 learned_addr learned my addr 172.21.15.100:0/477806134 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:08.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57de452640 10 monclient(hunting): get_auth_request con 0x7f57d806ea50 auth_method 0 2024-09-13T11:53:08.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57de452640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57de452640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57de452640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57ddc51640 10 monclient(hunting): get_auth_request con 0x7f57d806ee30 auth_method 0 2024-09-13T11:53:08.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57ddc51640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57ddc51640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57ddc51640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57de452640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:08.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57de452640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:08.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57de452640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:08.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9652ffd640 10 monclient(hunting): handle_auth_done global_id 4265 payload 824 2024-09-13T11:53:08.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f96537fe640 1 --2- 172.21.15.100:0/1271138643 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f965413c9f0 0x7f965409dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f9652ffd640 1 -- 172.21.15.100:0/1271138643 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f965413c9f0 msgr2=0x7f965409dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57dec53640 1 --2- 172.21.15.100:0/477806134 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57d806f790 0x7f57d81247c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57dec53640 10 monclient(hunting): get_auth_request con 0x7f57d806f790 auth_method 0 2024-09-13T11:53:08.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57dec53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57dec53640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.040+0000 7f57dec53640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57dec53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57dec53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57dec53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57ddc51640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57ddc51640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57ddc51640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57de452640 10 monclient(hunting): handle_auth_done global_id 4285 payload 824 2024-09-13T11:53:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57de452640 1 -- 172.21.15.100:0/477806134 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57d806f790 msgr2=0x7f57d81247c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57de452640 1 --2- 172.21.15.100:0/477806134 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57d806f790 0x7f57d81247c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f9652ffd640 1 --2- 172.21.15.100:0/1271138643 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f965413c9f0 0x7f965409dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f9652ffd640 1 -- 172.21.15.100:0/1271138643 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96540a2860 msgr2=0x7f9654137880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f9652ffd640 1 --2- 172.21.15.100:0/1271138643 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96540a2860 0x7f9654137880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f9652ffd640 10 monclient: _finish_hunting 0 2024-09-13T11:53:08.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f9652ffd640 1 monclient: found mon.noname-c 2024-09-13T11:53:08.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f9652ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:08.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f9652ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:53:08.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f9652ffd640 1 -- 172.21.15.100:0/1271138643 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96541401c0 con 0x7f965406ea50 2024-09-13T11:53:08.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f964bfff640 1 --2- 172.21.15.100:0/1271138643 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96540a2860 0x7f9654137880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:08.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f96537fe640 10 monclient: get_auth_request con 0x7f965413c9f0 auth_method 0 2024-09-13T11:53:08.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f96537fe640 1 --2- 172.21.15.100:0/1271138643 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f965413c9f0 0x7f965409dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:53:08.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57de452640 1 -- 172.21.15.100:0/477806134 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f57d806ee30 msgr2=0x7f57d8124280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57de452640 1 --2- 172.21.15.100:0/477806134 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f57d806ee30 0x7f57d8124280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57de452640 10 monclient: _finish_hunting 0 2024-09-13T11:53:08.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57de452640 1 monclient: found mon.noname-b 2024-09-13T11:53:08.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57de452640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:08.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57de452640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.122:3300/0 2024-09-13T11:53:08.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57de452640 1 -- 172.21.15.100:0/477806134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57d8127880 con 0x7f57d806ea50 2024-09-13T11:53:08.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57de452640 1 --2- 172.21.15.100:0/477806134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f57d806ea50 0x7f57d8123d40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f57d402a720 tx=0x7f57d40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:08.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f9652ffd640 1 --2- 172.21.15.100:0/1271138643 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f965406ea50 0x7f96541cc730 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f964400c440 tx=0x7f964400c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:08.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57ceffd640 1 -- 172.21.15.100:0/477806134 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d4041020 con 0x7f57d806ea50 2024-09-13T11:53:08.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:08.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57ceffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:08.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57ceffd640 10 monclient: dump: 2024-09-13T11:53:08.110 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:08.110 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.110 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.110 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.111 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:08.111 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.111 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:08.111 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:08.111 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:08.111 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.111 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57ceffd640 10 monclient: _finish_auth 0 2024-09-13T11:53:08.111 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57ceffd640 10 monclient: _check_auth_tickets 2024-09-13T11:53:08.111 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:08.112 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f964b7fe640 1 -- 172.21.15.100:0/1271138643 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9644017070 con 0x7f965406ea50 2024-09-13T11:53:08.112 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f964b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:08.112 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f964b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:08.112 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f964b7fe640 10 monclient: dump: 2024-09-13T11:53:08.112 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:08.112 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.112 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.112 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:08.113 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f964b7fe640 10 monclient: _finish_auth 0 2024-09-13T11:53:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f964b7fe640 10 monclient: _check_auth_tickets 2024-09-13T11:53:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f964b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f964b7fe640 1 -- 172.21.15.100:0/1271138643 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96440035f0 con 0x7f965406ea50 2024-09-13T11:53:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f964b7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f964b7fe640 1 -- 172.21.15.100:0/1271138643 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96440039e0 con 0x7f965406ea50 2024-09-13T11:53:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f964b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f964b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:08.114 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f964b7fe640 10 monclient: dump: 2024-09-13T11:53:08.114 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:08.114 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.114 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.114 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.114 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:08.114 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.114 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:08.114 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:08.114 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:08.114 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.115 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57ceffd640 1 -- 172.21.15.100:0/477806134 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57d4030760 con 0x7f57d806ea50 2024-09-13T11:53:08.115 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57ceffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:08.115 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57ceffd640 1 -- 172.21.15.100:0/477806134 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d4037040 con 0x7f57d806ea50 2024-09-13T11:53:08.115 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:08.115 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57ceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:53:08.115 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57ceffd640 10 monclient: dump: 2024-09-13T11:53:08.115 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:08.115 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.115 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.115 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.116 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:08.116 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.116 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:08.116 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:08.116 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:08.116 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.116 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57e06b6640 5 monclient: authenticate success, global_id 4285 2024-09-13T11:53:08.116 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57e06b6640 10 monclient: _renew_subs 2024-09-13T11:53:08.116 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57dcc4f640 4 set_mon_vals no callback set 2024-09-13T11:53:08.116 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57e06b6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:53:08.116 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57e06b6640 1 -- 172.21.15.100:0/477806134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57d8127b60 con 0x7f57d806ea50 2024-09-13T11:53:08.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f9659649640 5 monclient: authenticate success, global_id 4265 2024-09-13T11:53:08.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f9659649640 10 monclient: _renew_subs 2024-09-13T11:53:08.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f9659649640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:08.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57e06b6640 10 monclient: _renew_subs 2024-09-13T11:53:08.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57e06b6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:53:08.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f57e06b6640 1 -- 172.21.15.100:0/477806134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57d8126cf0 con 0x7f57d806ea50 2024-09-13T11:53:08.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f9659649640 1 -- 172.21.15.100:0/1271138643 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96540a10d0 con 0x7f965406ea50 2024-09-13T11:53:08.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f9659649640 10 monclient: _renew_subs 2024-09-13T11:53:08.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f9659649640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:08.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f9659649640 1 -- 172.21.15.100:0/1271138643 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9654120960 con 0x7f965406ea50 2024-09-13T11:53:08.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.041+0000 7f9651ffb640 4 set_mon_vals no callback set 2024-09-13T11:53:08.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.043+0000 7f57e06b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:08.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.043+0000 7f57e06b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:08.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.043+0000 7f57e06b6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:53:08.119 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.043+0000 7f57e06b6640 1 -- 172.21.15.100:0/477806134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57d80a2860 con 0x7f57d806ea50 2024-09-13T11:53:08.119 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.043+0000 7f9659649640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:08.119 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.043+0000 7f9659649640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:08.119 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.043+0000 7f9659649640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:08.119 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.043+0000 7f9659649640 1 -- 172.21.15.100:0/1271138643 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f965406ee30 con 0x7f965406ea50 2024-09-13T11:53:08.120 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.043+0000 7f57ceffd640 1 -- 172.21.15.100:0/477806134 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f57d4044020 con 0x7f57d806ea50 2024-09-13T11:53:08.120 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.043+0000 7f57ceffd640 1 --2- 172.21.15.100:0/477806134 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f57bc03dfa0 0x7f57bc040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.120 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.043+0000 7f57ceffd640 1 -- 172.21.15.100:0/477806134 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7f57d4072eb0 con 0x7f57d806ea50 2024-09-13T11:53:08.120 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.044+0000 7f57ddc51640 1 --2- 172.21.15.100:0/477806134 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f57bc03dfa0 0x7f57bc040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.120 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.044+0000 7f57ddc51640 10 monclient: get_auth_request con 0x7f57bc03dfa0 auth_method 0 2024-09-13T11:53:08.120 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.044+0000 7f57ddc51640 1 --2- 172.21.15.100:0/477806134 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f57bc03dfa0 0x7f57bc040460 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f57d8137590 tx=0x7f57c8008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:08.121 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.044+0000 7f964b7fe640 1 -- 172.21.15.100:0/1271138643 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f9644003b80 con 0x7f965406ea50 2024-09-13T11:53:08.121 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.044+0000 7f964b7fe640 1 --2- 172.21.15.100:0/1271138643 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f963403dfa0 0x7f9634040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.121 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.044+0000 7f964b7fe640 1 -- 172.21.15.100:0/1271138643 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7f9644051cf0 con 0x7f965406ea50 2024-09-13T11:53:08.121 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.045+0000 7f964bfff640 1 --2- 172.21.15.100:0/1271138643 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f963403dfa0 0x7f9634040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.122 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.045+0000 7f964bfff640 10 monclient: get_auth_request con 0x7f963403dfa0 auth_method 0 2024-09-13T11:53:08.122 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.045+0000 7f964bfff640 1 --2- 172.21.15.100:0/1271138643 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f963403dfa0 0x7f9634040460 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f963c034000 tx=0x7f963c02e000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:08.122 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.048+0000 7f964b7fe640 1 -- 172.21.15.100:0/1271138643 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f9644020080 con 0x7f965406ea50 2024-09-13T11:53:08.122 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.049+0000 7f964b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:08.122 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.049+0000 7f964b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:08.122 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.049+0000 7f57ceffd640 1 -- 172.21.15.100:0/477806134 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f57d404b030 con 0x7f57d806ea50 2024-09-13T11:53:08.122 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.049+0000 7f57ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:08.122 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.049+0000 7f57ceffd640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:08.125 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.069+0000 7f3f0b857640 10 monclient: get_monmap_and_config 2024-09-13T11:53:08.125 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.069+0000 7f3f0b857640 10 monclient: build_initial_monmap 2024-09-13T11:53:08.125 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.069+0000 7f3f0b857640 10 monclient: monmap: 2024-09-13T11:53:08.125 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:08.125 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.125 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:08.070520+0000 2024-09-13T11:53:08.125 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:08.070520+0000 2024-09-13T11:53:08.125 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:08.126 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.126 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:08.126 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:08.126 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:08.126 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:08.126 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:08.126 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.126 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.070+0000 7f3f0b857640 1 Processor -- start 2024-09-13T11:53:08.126 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.070+0000 7f3f0b857640 1 -- start start 2024-09-13T11:53:08.126 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.070+0000 7f3f0b857640 10 monclient: init 2024-09-13T11:53:08.126 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f0b857640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:08.126 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f0b857640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-13T11:53:08.126 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f0b857640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f04137a60 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:08.126 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f0b857640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3f040a2980 0x7f3f0413c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f0b857640 10 monclient(hunting): picked mon.noname-b con 0x7f3f040a2980 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:08.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f0b857640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3f0413ce90 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:08.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f0b857640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f04138550 con 0x7f3f0413ce90 2024-09-13T11:53:08.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f0b857640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f040a5be0 con 0x7f3f04137a60 2024-09-13T11:53:08.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f0b857640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f0b857640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f3f040a4410 con 0x7f3f040a2980 2024-09-13T11:53:08.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f0b857640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:08.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f0b857640 10 monclient(hunting): authenticate will time out at 768.664379s 2024-09-13T11:53:08.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f09df4640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f3f0413ce90 0x7f3f0413f280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:57996/0 (socket says 172.21.15.100:57996) 2024-09-13T11:53:08.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f09df4640 1 -- 172.21.15.100:0/541512533 learned_addr learned my addr 172.21.15.100:0/541512533 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:53:08.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f08df2640 1 --2- 172.21.15.100:0/541512533 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3f040a2980 0x7f3f0413c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f08df2640 10 monclient(hunting): get_auth_request con 0x7f3f040a2980 auth_method 0 2024-09-13T11:53:08.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f08df2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f08df2640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f08df2640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f08df2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:08.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f08df2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:08.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f08df2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:08.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f08df2640 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2024-09-13T11:53:08.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f08df2640 1 -- 172.21.15.100:0/541512533 >> v1:172.21.15.122:6789/0 conn(0x7f3f04137a60 legacy=0x7f3f04137e40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-13T11:53:08.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f08df2640 1 -- 172.21.15.100:0/541512533 >> v1:172.21.15.100:6789/0 conn(0x7f3f0413ce90 legacy=0x7f3f0413f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f08df2640 10 monclient: _finish_hunting 0 2024-09-13T11:53:08.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f08df2640 1 monclient: found mon.noname-b 2024-09-13T11:53:08.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3efbfff640 1 -- 172.21.15.100:0/541512533 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3048301303 0 0) 0x7f3f04138550 con 0x7f3f0413ce90 2024-09-13T11:53:08.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f08df2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:08.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f08df2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:53:08.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3f08df2640 1 -- 172.21.15.100:0/541512533 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f040a76b0 con 0x7f3f040a2980 2024-09-13T11:53:08.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.071+0000 7f3efbfff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:53:08.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.072+0000 7f3f08df2640 1 --2- 172.21.15.100:0/541512533 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3f040a2980 0x7f3f0413c950 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3ef40097d0 tx=0x7f3ef402a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e08c19eb99c6b90a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:08.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.072+0000 7f688fa40640 10 monclient: get_monmap_and_config 2024-09-13T11:53:08.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.072+0000 7f688fa40640 10 monclient: build_initial_monmap 2024-09-13T11:53:08.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.072+0000 7f688fa40640 10 monclient: monmap: 2024-09-13T11:53:08.129 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:08.129 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.129 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:08.073629+0000 2024-09-13T11:53:08.129 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:08.073629+0000 2024-09-13T11:53:08.129 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:08.130 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.130 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:08.130 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:08.130 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:08.130 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:08.130 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:08.130 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.072+0000 7f3efbfff640 1 -- 172.21.15.100:0/541512533 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4037070 con 0x7f3f040a2980 2024-09-13T11:53:08.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.072+0000 7f3efbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:08.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.072+0000 7f3efbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:08.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.072+0000 7f3efbfff640 10 monclient: dump: 2024-09-13T11:53:08.131 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:08.131 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.131 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.131 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.131 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:08.131 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.131 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:08.131 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:08.131 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:08.132 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.072+0000 7f3efbfff640 10 monclient: _finish_auth 0 2024-09-13T11:53:08.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.072+0000 7f3efbfff640 10 monclient: _check_auth_tickets 2024-09-13T11:53:08.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.072+0000 7f3efbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:08.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.072+0000 7f3f0b857640 5 monclient: authenticate success, global_id 4271 2024-09-13T11:53:08.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.072+0000 7f3f0b857640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:08.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.073+0000 7f3efbfff640 1 -- 172.21.15.100:0/541512533 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ef402ad90 con 0x7f3f040a2980 2024-09-13T11:53:08.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.073+0000 7f3efbfff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:08.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.073+0000 7f3f0b857640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:08.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.073+0000 7f3f0b857640 4 set_mon_vals no callback set 2024-09-13T11:53:08.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.073+0000 7f3efbfff640 1 -- 172.21.15.100:0/541512533 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4032660 con 0x7f3f040a2980 2024-09-13T11:53:08.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.073+0000 7f3efbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:08.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.073+0000 7f3efbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:08.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.073+0000 7f3efbfff640 10 monclient: dump: 2024-09-13T11:53:08.133 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:08.133 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.133 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.133 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.133 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:08.133 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.133 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:08.133 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:08.133 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:08.133 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.073+0000 7f688fa40640 1 Processor -- start 2024-09-13T11:53:08.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.073+0000 7f3f0b857640 10 monclient: shutdown 2024-09-13T11:53:08.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.073+0000 7f3f0b857640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:08.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.073+0000 7f3f0b857640 1 -- 172.21.15.100:0/541512533 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3f040a2980 msgr2=0x7f3f0413c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.073+0000 7f3f0b857640 1 --2- 172.21.15.100:0/541512533 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3f040a2980 0x7f3f0413c950 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3ef40097d0 tx=0x7f3ef402a040 comp rx=0 tx=0).stop 2024-09-13T11:53:08.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.073+0000 7f3f0b857640 1 -- 172.21.15.100:0/541512533 shutdown_connections 2024-09-13T11:53:08.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.073+0000 7f3f0b857640 1 --2- 172.21.15.100:0/541512533 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3f040a2980 0x7f3f0413c950 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3ef40097d0 tx=0x7f3ef402a040 comp rx=0 tx=0).stop 2024-09-13T11:53:08.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.073+0000 7f3f0b857640 1 -- 172.21.15.100:0/541512533 >> 172.21.15.100:0/541512533 conn(0x7f3f040aa580 msgr2=0x7f3f040a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:08.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.073+0000 7fc76162a640 10 monclient: get_monmap_and_config 2024-09-13T11:53:08.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.073+0000 7fc76162a640 10 monclient: build_initial_monmap 2024-09-13T11:53:08.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.073+0000 7fc76162a640 10 monclient: monmap: 2024-09-13T11:53:08.135 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:08.135 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.135 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:08.074801+0000 2024-09-13T11:53:08.135 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:08.074801+0000 2024-09-13T11:53:08.135 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:08.135 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.135 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:08.135 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:08.135 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:08.136 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:08.136 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:08.136 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f3f0b857640 1 -- 172.21.15.100:0/541512533 shutdown_connections 2024-09-13T11:53:08.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f3f0b857640 1 -- 172.21.15.100:0/541512533 wait complete. 2024-09-13T11:53:08.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.073+0000 7f688fa40640 1 -- start start 2024-09-13T11:53:08.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.073+0000 7f688fa40640 10 monclient: init 2024-09-13T11:53:08.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7fc76162a640 1 Processor -- start 2024-09-13T11:53:08.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688fa40640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:08.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.075+0000 7f3f0b857640 10 monclient: build_initial_monmap 2024-09-13T11:53:08.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.075+0000 7f3f0b857640 10 monclient: monmap: 2024-09-13T11:53:08.137 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:08.138 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:08.138 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:08.138 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:08.138 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:08.138 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.138 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:08.138 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:08.138 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:08.138 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7fc76162a640 1 -- start start 2024-09-13T11:53:08.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.075+0000 7fc76162a640 10 monclient: init 2024-09-13T11:53:08.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688fa40640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-13T11:53:08.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688fa40640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f688806ea50 0x7f688806ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688fa40640 10 monclient(hunting): picked mon.noname-c con 0x7f688806ea50 addr v2:172.21.15.122:3300/0 2024-09-13T11:53:08.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688fa40640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f688806f370 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:08.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688fa40640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f688813c480 0x7f6888143fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688fa40640 10 monclient(hunting): picked mon.noname-b con 0x7f688813c480 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:08.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688fa40640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6888074d30 con 0x7f688806f370 2024-09-13T11:53:08.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688dfdd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f688813c480 0x7f6888143fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688dfdd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f688813c480 0x7f6888143fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47378/0 (socket says 172.21.15.100:47378) 2024-09-13T11:53:08.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688dfdd640 1 -- 172.21.15.100:0/774325245 learned_addr learned my addr 172.21.15.100:0/774325245 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:08.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688fa40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688fa40640 1 -- 172.21.15.100:0/774325245 --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f68880a1870 con 0x7f688806ea50 2024-09-13T11:53:08.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688fa40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688fa40640 1 -- 172.21.15.100:0/774325245 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f68880708d0 con 0x7f688813c480 2024-09-13T11:53:08.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688fa40640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:08.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688fa40640 10 monclient(hunting): authenticate will time out at 768.667669s 2024-09-13T11:53:08.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688dfdd640 10 monclient(hunting): get_auth_request con 0x7f688813c480 auth_method 0 2024-09-13T11:53:08.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688dfdd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688dfdd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688dfdd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688d7dc640 1 --2- 172.21.15.100:0/774325245 >> v2:172.21.15.122:3300/0 conn(0x7f688806ea50 0x7f688806ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688d7dc640 10 monclient(hunting): get_auth_request con 0x7f688806ea50 auth_method 0 2024-09-13T11:53:08.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688d7dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688d7dc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688d7dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688dfdd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:08.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688dfdd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:08.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688dfdd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:08.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688d7dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:08.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688d7dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:08.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688d7dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:08.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688dfdd640 10 monclient(hunting): handle_auth_done global_id 4274 payload 274 2024-09-13T11:53:08.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688dfdd640 1 -- 172.21.15.100:0/774325245 >> v2:172.21.15.122:3300/0 conn(0x7f688806ea50 msgr2=0x7f688806ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:08.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688dfdd640 1 --2- 172.21.15.100:0/774325245 >> v2:172.21.15.122:3300/0 conn(0x7f688806ea50 0x7f688806ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688dfdd640 1 -- 172.21.15.100:0/774325245 >> v1:172.21.15.122:6789/0 conn(0x7f688806f370 legacy=0x7f688813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:08.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688dfdd640 10 monclient: _finish_hunting 0 2024-09-13T11:53:08.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688dfdd640 1 monclient: found mon.noname-b 2024-09-13T11:53:08.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688dfdd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:08.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688dfdd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:53:08.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.074+0000 7f688dfdd640 1 -- 172.21.15.100:0/774325245 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68880a3f70 con 0x7f688813c480 2024-09-13T11:53:08.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.075+0000 7f3f0b857640 1 Processor -- start 2024-09-13T11:53:08.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.075+0000 7f3f0b857640 1 -- start start 2024-09-13T11:53:08.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.075+0000 7f3f0b857640 10 monclient: init 2024-09-13T11:53:08.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.075+0000 7fc76162a640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:08.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.075+0000 7fc76162a640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-13T11:53:08.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.075+0000 7fc76162a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc75c06ea50 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:08.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.075+0000 7fc76162a640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc75c0a2860 0x7fc75c06f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.075+0000 7fc76162a640 10 monclient(hunting): picked mon.noname-a con 0x7fc75c0a2860 addr v2:172.21.15.100:3300/0 2024-09-13T11:53:08.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.075+0000 7fc76162a640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc75c13c9f0 0x7fc75c13ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc75affd640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fc75c06ea50 0x7fc75c06ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:58002/0 (socket says 172.21.15.100:58002) 2024-09-13T11:53:08.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.075+0000 7fb84db2b640 10 monclient: get_monmap_and_config 2024-09-13T11:53:08.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.075+0000 7fb84db2b640 10 monclient: build_initial_monmap 2024-09-13T11:53:08.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.075+0000 7fb84db2b640 10 monclient: monmap: 2024-09-13T11:53:08.144 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:08.144 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.144 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:08.076811+0000 2024-09-13T11:53:08.144 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:08.076811+0000 2024-09-13T11:53:08.144 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:08.144 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.144 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:08.144 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:08.144 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:08.144 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:08.144 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:08.145 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.145 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.075+0000 7f688dfdd640 1 --2- 172.21.15.100:0/774325245 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f688813c480 0x7f6888143fd0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6880009e50 tx=0x7f688002a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7874f9a2adb898c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:08.145 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc75a7fc640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc75c0a2860 0x7fc75c06f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.145 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f687ffff640 1 -- 172.21.15.100:0/774325245 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6880036070 con 0x7f688813c480 2024-09-13T11:53:08.145 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f687ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:08.145 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f687ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:08.146 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc75affd640 1 -- 172.21.15.100:0/1433512282 learned_addr learned my addr 172.21.15.100:0/1433512282 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:53:08.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc76162a640 10 monclient(hunting): picked mon.noname-b con 0x7fc75c13c9f0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:08.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc76162a640 1 -- 172.21.15.100:0/1433512282 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc75c06fa80 con 0x7fc75c06ea50 2024-09-13T11:53:08.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc76162a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc76162a640 1 -- 172.21.15.100:0/1433512282 --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7fc75c0a2cb0 con 0x7fc75c0a2860 2024-09-13T11:53:08.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc76162a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc76162a640 1 -- 172.21.15.100:0/1433512282 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fc75c0a15e0 con 0x7fc75c13c9f0 2024-09-13T11:53:08.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc76162a640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:08.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc76162a640 10 monclient(hunting): authenticate will time out at 768.669473s 2024-09-13T11:53:08.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f687ffff640 10 monclient: dump: 2024-09-13T11:53:08.147 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:08.147 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.148 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.148 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.148 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:08.148 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.148 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:08.148 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:08.148 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:08.148 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f687ffff640 10 monclient: _finish_auth 0 2024-09-13T11:53:08.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f687ffff640 10 monclient: _check_auth_tickets 2024-09-13T11:53:08.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f687ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:08.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f687ffff640 1 -- 172.21.15.100:0/774325245 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6880031600 con 0x7f688813c480 2024-09-13T11:53:08.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f688fa40640 5 monclient: authenticate success, global_id 4274 2024-09-13T11:53:08.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f688fa40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:08.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f3f0b857640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:08.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f3f0b857640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:53:08.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f3f0b857640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f04137a60 0x7f3f041365c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f3f0b857640 10 monclient(hunting): picked mon.noname-b con 0x7f3f04137a60 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:08.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f3f0b857640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3f0413ce90 0x7f3f0409ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f3f0b857640 10 monclient(hunting): picked mon.noname-a con 0x7f3f0413ce90 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:08.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f3f0b857640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3f04136b00 0x7f3f0409cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f3f0b857640 10 monclient(hunting): picked mon.noname-c con 0x7f3f04136b00 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:08.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f3f0b857640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f3f0b857640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0406fd80 con 0x7f3f0413ce90 2024-09-13T11:53:08.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f3f0b857640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f3f0b857640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f3f040a1700 con 0x7f3f04137a60 2024-09-13T11:53:08.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f3f0b857640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f3f0b857640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f3f0406ff00 con 0x7f3f04136b00 2024-09-13T11:53:08.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f3f0b857640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:08.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc75b7fe640 1 --2- 172.21.15.100:0/1433512282 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc75c13c9f0 0x7fc75c13ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc75a7fc640 10 monclient(hunting): get_auth_request con 0x7fc75c0a2860 auth_method 0 2024-09-13T11:53:08.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f3f0b857640 10 monclient(hunting): authenticate will time out at 768.669608s 2024-09-13T11:53:08.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc75a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc75a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc75a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc75b7fe640 10 monclient(hunting): get_auth_request con 0x7fc75c13c9f0 auth_method 0 2024-09-13T11:53:08.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc75b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc75b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc75b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc759ffb640 1 -- 172.21.15.100:0/1433512282 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3425273643 0 0) 0x7fc750001110 con 0x7fc75c06ea50 2024-09-13T11:53:08.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc759ffb640 10 monclient(hunting): _negotiate 2024-09-13T11:53:08.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc759ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f3f09df4640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3f04136b00 0x7f3f0409cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc759ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc759ffb640 10 monclient(hunting): my global_id is 4311 2024-09-13T11:53:08.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fb84db2b640 1 Processor -- start 2024-09-13T11:53:08.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc759ffb640 1 -- 172.21.15.100:0/1433512282 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc744003520 con 0x7fc75c06ea50 2024-09-13T11:53:08.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f3f09df4640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3f04136b00 0x7f3f0409cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47382/0 (socket says 172.21.15.100:47382) 2024-09-13T11:53:08.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc75a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:08.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fc75a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:08.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fc75a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:08.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f3f09df4640 1 -- 172.21.15.100:0/1462170745 learned_addr learned my addr 172.21.15.100:0/1462170745 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:08.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fc759ffb640 1 -- 172.21.15.100:0/1433512282 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3495740310 0 0) 0x7fc744003520 con 0x7fc75c06ea50 2024-09-13T11:53:08.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fc75a7fc640 10 monclient(hunting): handle_auth_done global_id 4314 payload 274 2024-09-13T11:53:08.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fc75a7fc640 1 -- 172.21.15.100:0/1433512282 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc75c13c9f0 msgr2=0x7fc75c13ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:08.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fc75a7fc640 1 --2- 172.21.15.100:0/1433512282 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc75c13c9f0 0x7fc75c13ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fc75a7fc640 1 -- 172.21.15.100:0/1433512282 >> v1:172.21.15.100:6789/0 conn(0x7fc75c06ea50 legacy=0x7fc75c06ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fc75a7fc640 10 monclient: _finish_hunting 0 2024-09-13T11:53:08.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fc75a7fc640 1 monclient: found mon.noname-a 2024-09-13T11:53:08.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fc75a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:08.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fc75a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:08.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fc75a7fc640 1 -- 172.21.15.100:0/1433512282 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc75c0a1870 con 0x7fc75c0a2860 2024-09-13T11:53:08.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fc759ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:53:08.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f3f09df4640 10 monclient(hunting): get_auth_request con 0x7f3f04136b00 auth_method 0 2024-09-13T11:53:08.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f3f09df4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f3f09df4640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7f3f09df4640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7f3f08df2640 1 --2- 172.21.15.100:0/1462170745 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3f0413ce90 0x7f3f0409ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7f3f09df4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:08.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7f3f09df4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:08.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7f3f09df4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:08.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7f3f08df2640 10 monclient(hunting): get_auth_request con 0x7f3f0413ce90 auth_method 0 2024-09-13T11:53:08.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7f3f08df2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7f3f08df2640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7f3f08df2640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7f3f08df2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:08.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7f3f08df2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:08.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7f3f08df2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:08.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7f3f09df4640 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2024-09-13T11:53:08.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fc75b7fe640 1 --2- 172.21.15.100:0/1433512282 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc75c13c9f0 0x7fc75c13ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:53:08.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7f3f09df4640 1 -- 172.21.15.100:0/1462170745 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f04137a60 msgr2=0x7f3f041365c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-13T11:53:08.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7f3f09df4640 1 --2- 172.21.15.100:0/1462170745 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f04137a60 0x7f3f041365c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7f3f09df4640 1 -- 172.21.15.100:0/1462170745 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3f0413ce90 msgr2=0x7f3f0409ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7f3f09df4640 1 --2- 172.21.15.100:0/1462170745 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3f0413ce90 0x7f3f0409ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7f3f09df4640 10 monclient: _finish_hunting 0 2024-09-13T11:53:08.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7f3f09df4640 1 monclient: found mon.noname-c 2024-09-13T11:53:08.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7f3f09df4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:08.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7f3f09df4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:53:08.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7f3f09df4640 1 -- 172.21.15.100:0/1462170745 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f040a0600 con 0x7f3f04136b00 2024-09-13T11:53:08.159 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fb84db2b640 1 -- start start 2024-09-13T11:53:08.159 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.076+0000 7fb84db2b640 10 monclient: init 2024-09-13T11:53:08.159 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fb84db2b640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:08.159 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7f3f08df2640 1 --2- 172.21.15.100:0/1462170745 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3f0413ce90 0x7f3f0409ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:08.159 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fb84db2b640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-13T11:53:08.160 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fb84db2b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb84806ea50 0x7fb84806ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.160 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fb84db2b640 10 monclient(hunting): picked mon.noname-b con 0x7fb84806ea50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:08.160 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fb84db2b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb84806f370 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:08.160 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fb84db2b640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb84813c480 0x7fb848143fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.160 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7f3f09df4640 1 --2- 172.21.15.100:0/1462170745 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3f04136b00 0x7f3f0409cf90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3efc00c7f0 tx=0x7f3efc00ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:08.161 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fb84db2b640 10 monclient(hunting): picked mon.noname-a con 0x7fb84813c480 addr v2:172.21.15.100:3300/0 2024-09-13T11:53:08.161 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fb84db2b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb848074d30 con 0x7fb84806f370 2024-09-13T11:53:08.161 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fb84db2b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.161 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fb84db2b640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7fb8480a1870 con 0x7fb84813c480 2024-09-13T11:53:08.161 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fb84db2b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.161 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fb84db2b640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fb8480708d0 con 0x7fb84806ea50 2024-09-13T11:53:08.161 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fb84db2b640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:08.161 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fb84db2b640 10 monclient(hunting): authenticate will time out at 768.670766s 2024-09-13T11:53:08.161 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fb8477fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb84806ea50 0x7fb84806ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.161 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7f3ef9ffb640 1 -- 172.21.15.100:0/1462170745 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3efc018070 con 0x7f3f04136b00 2024-09-13T11:53:08.162 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fb8477fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb84806ea50 0x7fb84806ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47392/0 (socket says 172.21.15.100:47392) 2024-09-13T11:53:08.162 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fb8477fe640 1 -- 172.21.15.100:0/2292264902 learned_addr learned my addr 172.21.15.100:0/2292264902 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:08.162 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fb8477fe640 10 monclient(hunting): get_auth_request con 0x7fb84806ea50 auth_method 0 2024-09-13T11:53:08.162 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fb8477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.162 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fb8477fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.162 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7f3ef9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:08.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fb8477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.077+0000 7fb846ffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb84806f370 0x7fb84813bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.100:52474/0 (socket says 172.21.15.100:52474) 2024-09-13T11:53:08.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7f3ef9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:08.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fb8477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:08.164 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7f3ef9ffb640 10 monclient: dump: 2024-09-13T11:53:08.164 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:08.164 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.164 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.164 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.164 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:08.164 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.164 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:08.164 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:08.164 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:08.164 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.165 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fb8477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:08.165 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7f3ef9ffb640 10 monclient: _finish_auth 0 2024-09-13T11:53:08.165 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7f3ef9ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:53:08.165 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7f3ef9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:08.165 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fb8477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:08.166 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7f3f0b857640 5 monclient: authenticate success, global_id 4280 2024-09-13T11:53:08.166 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7f3f0b857640 10 monclient: _renew_subs 2024-09-13T11:53:08.166 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7f3f0b857640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:08.166 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fc75a7fc640 1 --2- 172.21.15.100:0/1433512282 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc75c0a2860 0x7fc75c06f540 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc74c0097d0 tx=0x7fc74c02a020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79e293c8ad20585f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:08.167 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7f3f0b857640 1 -- 172.21.15.100:0/1462170745 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f040a1140 con 0x7f3f04136b00 2024-09-13T11:53:08.167 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7f3ef9ffb640 1 -- 172.21.15.100:0/1462170745 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3efc00a640 con 0x7f3f04136b00 2024-09-13T11:53:08.167 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fb8477fe640 10 monclient(hunting): handle_auth_done global_id 4283 payload 274 2024-09-13T11:53:08.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fb8477fe640 1 -- 172.21.15.100:0/2292264902 >> v2:172.21.15.100:3300/0 conn(0x7fb84813c480 msgr2=0x7fb848143fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:08.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fb847fff640 1 --2- 172.21.15.100:0/2292264902 >> v2:172.21.15.100:3300/0 conn(0x7fb84813c480 0x7fb848143fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fb8477fe640 1 --2- 172.21.15.100:0/2292264902 >> v2:172.21.15.100:3300/0 conn(0x7fb84813c480 0x7fb848143fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fb8477fe640 1 -- 172.21.15.100:0/2292264902 >> v1:172.21.15.122:6789/0 conn(0x7fb84806f370 legacy=0x7fb84813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:08.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fb8477fe640 10 monclient: _finish_hunting 0 2024-09-13T11:53:08.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fb8477fe640 1 monclient: found mon.noname-b 2024-09-13T11:53:08.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7f3ef9ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:08.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7f3ef9ffb640 1 -- 172.21.15.100:0/1462170745 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3efc00aa10 con 0x7f3f04136b00 2024-09-13T11:53:08.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7f3ef9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:08.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fb8477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:08.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fb8477fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:53:08.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7f3ef9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:08.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7f3ef9ffb640 10 monclient: dump: 2024-09-13T11:53:08.169 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:08.169 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.169 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.169 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.169 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:08.170 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.170 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:08.170 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:08.170 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:08.170 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.170 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fb8477fe640 1 -- 172.21.15.100:0/2292264902 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8480a3f70 con 0x7fb84806ea50 2024-09-13T11:53:08.170 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7f3efb7fe640 4 set_mon_vals no callback set 2024-09-13T11:53:08.171 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fb847fff640 10 monclient: get_auth_request con 0x7fb84813c480 auth_method 0 2024-09-13T11:53:08.171 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fb847fff640 1 --2- 172.21.15.100:0/2292264902 >> v2:172.21.15.100:3300/0 conn(0x7fb84813c480 0x7fb848143fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:53:08.171 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7f3f0b857640 10 monclient: _renew_subs 2024-09-13T11:53:08.171 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7f3f0b857640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:08.171 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7f3f0b857640 1 -- 172.21.15.100:0/1462170745 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f0409f240 con 0x7f3f04136b00 2024-09-13T11:53:08.173 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fc759ffb640 1 -- 172.21.15.100:0/1433512282 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc74c037070 con 0x7fc75c0a2860 2024-09-13T11:53:08.173 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fc759ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:08.173 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fc759ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:08.173 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fc759ffb640 10 monclient: dump: 2024-09-13T11:53:08.173 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:08.173 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.173 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.173 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.173 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:08.173 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.173 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:08.174 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:08.174 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:08.174 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fc759ffb640 10 monclient: _finish_auth 0 2024-09-13T11:53:08.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fc759ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:53:08.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fc759ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:08.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fc759ffb640 1 -- 172.21.15.100:0/1433512282 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc74c02aab0 con 0x7fc75c0a2860 2024-09-13T11:53:08.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fc759ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:08.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fc759ffb640 1 -- 172.21.15.100:0/1433512282 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7500018c0 con 0x7fc75c0a2860 2024-09-13T11:53:08.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fc759ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:08.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fc759ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:08.175 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.079+0000 7f3ef9ffb640 1 -- 172.21.15.100:0/1462170745 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f3efc01b070 con 0x7f3f04136b00 2024-09-13T11:53:08.175 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fc759ffb640 10 monclient: dump: 2024-09-13T11:53:08.175 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:08.175 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.175 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.175 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.175 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:08.175 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.175 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:08.175 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:08.175 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:08.176 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.176 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.079+0000 7fb8477fe640 1 --2- 172.21.15.100:0/2292264902 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb84806ea50 0x7fb84806ee30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb838009870 tx=0x7fb83802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8728f5510761b299 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:08.176 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fc76162a640 5 monclient: authenticate success, global_id 4314 2024-09-13T11:53:08.176 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fc76162a640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:08.177 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.079+0000 7f687ffff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:08.177 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.079+0000 7f688fa40640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:08.177 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.079+0000 7f688fa40640 4 set_mon_vals no callback set 2024-09-13T11:53:08.177 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.079+0000 7f687ffff640 1 -- 172.21.15.100:0/774325245 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68800319d0 con 0x7f688813c480 2024-09-13T11:53:08.177 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.079+0000 7f687ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:08.177 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.079+0000 7f687ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:08.177 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.079+0000 7f687ffff640 10 monclient: dump: 2024-09-13T11:53:08.177 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:08.177 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.177 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.177 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.177 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:08.177 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.177 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:08.178 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:08.178 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:08.178 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.178 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7f3ef9ffb640 1 --2- 172.21.15.100:0/1462170745 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f3ee803dfa0 0x7f3ee8040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.178 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.078+0000 7fc76162a640 4 set_mon_vals no callback set 2024-09-13T11:53:08.179 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7f3f0b857640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:08.179 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7f3f0b857640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:08.179 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7f3f0b857640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:08.179 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7f3f0b857640 1 -- 172.21.15.100:0/1462170745 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f04137e40 con 0x7f3f04136b00 2024-09-13T11:53:08.179 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.079+0000 7fc76162a640 10 monclient: shutdown 2024-09-13T11:53:08.179 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.079+0000 7fc76162a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:08.179 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.079+0000 7fc76162a640 1 -- 172.21.15.100:0/1433512282 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc75c0a2860 msgr2=0x7fc75c06f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.180 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.079+0000 7fc76162a640 1 --2- 172.21.15.100:0/1433512282 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc75c0a2860 0x7fc75c06f540 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc74c0097d0 tx=0x7fc74c02a020 comp rx=0 tx=0).stop 2024-09-13T11:53:08.180 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.079+0000 7fc76162a640 1 -- 172.21.15.100:0/1433512282 shutdown_connections 2024-09-13T11:53:08.180 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.079+0000 7fc76162a640 1 --2- 172.21.15.100:0/1433512282 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc75c0a2860 0x7fc75c06f540 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.180 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.079+0000 7fc76162a640 1 --2- 172.21.15.100:0/1433512282 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc75c13c9f0 0x7fc75c13ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.180 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.079+0000 7fc76162a640 1 -- 172.21.15.100:0/1433512282 >> 172.21.15.100:0/1433512282 conn(0x7fc75c0a68e0 msgr2=0x7fc75c0a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:08.180 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.079+0000 7fc76162a640 1 -- 172.21.15.100:0/1433512282 shutdown_connections 2024-09-13T11:53:08.180 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.079+0000 7fc76162a640 1 -- 172.21.15.100:0/1433512282 wait complete. 2024-09-13T11:53:08.180 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7f3ef9ffb640 1 -- 172.21.15.100:0/1462170745 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7f3efc051c90 con 0x7f3f04136b00 2024-09-13T11:53:08.181 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7f3f095f3640 1 --2- 172.21.15.100:0/1462170745 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f3ee803dfa0 0x7f3ee8040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.181 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7f3f095f3640 10 monclient: get_auth_request con 0x7f3ee803dfa0 auth_method 0 2024-09-13T11:53:08.181 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7fb8467fc640 1 -- 172.21.15.100:0/2292264902 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb838037070 con 0x7fb84806ea50 2024-09-13T11:53:08.181 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7f688fa40640 10 monclient: shutdown 2024-09-13T11:53:08.181 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7f688fa40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:08.181 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7f688fa40640 1 -- 172.21.15.100:0/774325245 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f688813c480 msgr2=0x7f6888143fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.182 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7f688fa40640 1 --2- 172.21.15.100:0/774325245 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f688813c480 0x7f6888143fd0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6880009e50 tx=0x7f688002a560 comp rx=0 tx=0).stop 2024-09-13T11:53:08.182 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7fb8467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:08.182 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7fb8467fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:08.182 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7fb8467fc640 10 monclient: dump: 2024-09-13T11:53:08.182 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:08.182 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.182 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.182 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.182 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:08.182 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.182 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:08.183 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:08.183 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:08.183 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.183 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7fb8467fc640 10 monclient: _finish_auth 0 2024-09-13T11:53:08.183 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7fb8467fc640 10 monclient: _check_auth_tickets 2024-09-13T11:53:08.183 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7fc76162a640 10 monclient: build_initial_monmap 2024-09-13T11:53:08.183 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7fb8467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:08.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7fb8467fc640 1 -- 172.21.15.100:0/2292264902 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb83802adf0 con 0x7fb84806ea50 2024-09-13T11:53:08.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7fb8467fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:08.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7fb8467fc640 1 -- 172.21.15.100:0/2292264902 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb838032660 con 0x7fb84806ea50 2024-09-13T11:53:08.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7fb8467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:08.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7fb8467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:08.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7fb8467fc640 10 monclient: dump: 2024-09-13T11:53:08.184 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:08.184 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.184 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.184 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.184 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:08.184 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.184 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:08.185 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:08.185 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:08.185 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7fb84db2b640 5 monclient: authenticate success, global_id 4283 2024-09-13T11:53:08.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7fb84db2b640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:08.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7fb84db2b640 4 set_mon_vals no callback set 2024-09-13T11:53:08.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7fc76162a640 10 monclient: monmap: 2024-09-13T11:53:08.185 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:08.185 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:08.185 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:08.185 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:08.186 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:08.186 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.186 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:08.186 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:08.186 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:08.186 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7f688fa40640 1 -- 172.21.15.100:0/774325245 shutdown_connections 2024-09-13T11:53:08.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7f688fa40640 1 --2- 172.21.15.100:0/774325245 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f688813c480 0x7f6888143fd0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7f688fa40640 1 --2- 172.21.15.100:0/774325245 >> v2:172.21.15.122:3300/0 conn(0x7f688806ea50 0x7f688806ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.080+0000 7f688fa40640 1 -- 172.21.15.100:0/774325245 >> 172.21.15.100:0/774325245 conn(0x7f68880a68e0 msgr2=0x7f68880a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:08.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.081+0000 7f688fa40640 1 -- 172.21.15.100:0/774325245 shutdown_connections 2024-09-13T11:53:08.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.081+0000 7fb84db2b640 10 monclient: shutdown 2024-09-13T11:53:08.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.081+0000 7fb84db2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:08.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.081+0000 7fb84db2b640 1 -- 172.21.15.100:0/2292264902 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb84806ea50 msgr2=0x7fb84806ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.081+0000 7fb84db2b640 1 --2- 172.21.15.100:0/2292264902 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb84806ea50 0x7fb84806ee30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb838009870 tx=0x7fb83802a0a0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.081+0000 7fb84db2b640 1 -- 172.21.15.100:0/2292264902 shutdown_connections 2024-09-13T11:53:08.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.081+0000 7fb84db2b640 1 --2- 172.21.15.100:0/2292264902 >> v2:172.21.15.100:3300/0 conn(0x7fb84813c480 0x7fb848143fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.081+0000 7fb84db2b640 1 --2- 172.21.15.100:0/2292264902 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb84806ea50 0x7fb84806ee30 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.081+0000 7fb84db2b640 1 -- 172.21.15.100:0/2292264902 >> 172.21.15.100:0/2292264902 conn(0x7fb8480a68e0 msgr2=0x7fb8480a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:08.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.081+0000 7f688fa40640 1 -- 172.21.15.100:0/774325245 wait complete. 2024-09-13T11:53:08.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.081+0000 7fc76162a640 1 Processor -- start 2024-09-13T11:53:08.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.081+0000 7fc76162a640 1 -- start start 2024-09-13T11:53:08.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.081+0000 7fc76162a640 10 monclient: init 2024-09-13T11:53:08.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.081+0000 7f3f095f3640 1 --2- 172.21.15.100:0/1462170745 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f3ee803dfa0 0x7f3ee8040460 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3f041377e0 tx=0x7f3f00006bd0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:08.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.081+0000 7fb84db2b640 1 -- 172.21.15.100:0/2292264902 shutdown_connections 2024-09-13T11:53:08.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fb84db2b640 1 -- 172.21.15.100:0/2292264902 wait complete. 2024-09-13T11:53:08.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7f688fa40640 10 monclient: build_initial_monmap 2024-09-13T11:53:08.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7f688fa40640 10 monclient: monmap: 2024-09-13T11:53:08.190 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:08.190 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:08.190 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:08.190 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:08.190 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:08.190 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.190 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:08.190 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:08.190 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:08.190 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc76162a640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:08.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc76162a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:53:08.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc76162a640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc75c06ea50 0x7fc75c1376e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc76162a640 10 monclient(hunting): picked mon.noname-a con 0x7fc75c06ea50 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:08.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc76162a640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc75c0a2860 0x7fc75c09ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75a7fc640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc75c0a2860 0x7fc75c09ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75a7fc640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc75c0a2860 0x7fc75c09ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47408/0 (socket says 172.21.15.100:47408) 2024-09-13T11:53:08.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75a7fc640 1 -- 172.21.15.100:0/1955712146 learned_addr learned my addr 172.21.15.100:0/1955712146 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:08.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc76162a640 10 monclient(hunting): picked mon.noname-c con 0x7fc75c0a2860 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:08.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc76162a640 1 --2- 172.21.15.100:0/1955712146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc75c13c9f0 0x7fc75c09f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75affd640 1 --2- 172.21.15.100:0/1955712146 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc75c06ea50 0x7fc75c1376e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc76162a640 10 monclient(hunting): picked mon.noname-b con 0x7fc75c13c9f0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:08.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fb84db2b640 10 monclient: build_initial_monmap 2024-09-13T11:53:08.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fb84db2b640 10 monclient: monmap: 2024-09-13T11:53:08.193 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:08.193 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:08.193 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:08.193 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:08.193 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:08.193 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.193 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:08.193 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:08.193 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:08.193 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc76162a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc76162a640 1 -- 172.21.15.100:0/1955712146 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fc75c0a5be0 con 0x7fc75c06ea50 2024-09-13T11:53:08.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc76162a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc76162a640 1 -- 172.21.15.100:0/1955712146 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc75c0a5a60 con 0x7fc75c13c9f0 2024-09-13T11:53:08.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc76162a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc76162a640 1 -- 172.21.15.100:0/1955712146 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fc75c13f320 con 0x7fc75c0a2860 2024-09-13T11:53:08.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc76162a640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:08.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7f688fa40640 1 Processor -- start 2024-09-13T11:53:08.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc76162a640 10 monclient(hunting): authenticate will time out at 768.675739s 2024-09-13T11:53:08.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75a7fc640 10 monclient(hunting): get_auth_request con 0x7fc75c0a2860 auth_method 0 2024-09-13T11:53:08.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75affd640 10 monclient(hunting): get_auth_request con 0x7fc75c06ea50 auth_method 0 2024-09-13T11:53:08.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75b7fe640 1 --2- 172.21.15.100:0/1955712146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc75c13c9f0 0x7fc75c09f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75b7fe640 10 monclient(hunting): get_auth_request con 0x7fc75c13c9f0 auth_method 0 2024-09-13T11:53:08.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:08.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:08.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:08.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:08.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:08.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:08.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:08.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:08.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7f688fa40640 1 -- start start 2024-09-13T11:53:08.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.082+0000 7fc75b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:08.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc75affd640 10 monclient(hunting): handle_auth_done global_id 4320 payload 824 2024-09-13T11:53:08.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7f688fa40640 10 monclient: init 2024-09-13T11:53:08.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc75affd640 1 -- 172.21.15.100:0/1955712146 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc75c0a2860 msgr2=0x7fc75c09ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc75affd640 1 --2- 172.21.15.100:0/1955712146 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc75c0a2860 0x7fc75c09ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fb84db2b640 1 Processor -- start 2024-09-13T11:53:08.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc75affd640 1 -- 172.21.15.100:0/1955712146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc75c13c9f0 msgr2=0x7fc75c09f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc75affd640 1 --2- 172.21.15.100:0/1955712146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc75c13c9f0 0x7fc75c09f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc75affd640 10 monclient: _finish_hunting 0 2024-09-13T11:53:08.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc75affd640 1 monclient: found mon.noname-a 2024-09-13T11:53:08.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc75affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:08.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc75affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:08.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc75affd640 1 -- 172.21.15.100:0/1955712146 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc75c0a0540 con 0x7fc75c06ea50 2024-09-13T11:53:08.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc75b7fe640 1 --2- 172.21.15.100:0/1955712146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc75c13c9f0 0x7fc75c09f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:08.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc75affd640 1 --2- 172.21.15.100:0/1955712146 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc75c06ea50 0x7fc75c1376e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc7500013f0 tx=0x7fc75000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:08.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc7437fe640 1 -- 172.21.15.100:0/1955712146 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc750017070 con 0x7fc75c06ea50 2024-09-13T11:53:08.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc7437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:08.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc7437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:08.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc7437fe640 10 monclient: dump: 2024-09-13T11:53:08.201 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:08.201 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.201 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.201 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.201 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:08.201 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.201 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:08.201 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:08.201 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:08.202 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc7437fe640 10 monclient: _finish_auth 0 2024-09-13T11:53:08.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc7437fe640 10 monclient: _check_auth_tickets 2024-09-13T11:53:08.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7f688fa40640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:08.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc7437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:08.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc7437fe640 1 -- 172.21.15.100:0/1955712146 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc750005ad0 con 0x7fc75c06ea50 2024-09-13T11:53:08.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc7437fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:08.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc7437fe640 1 -- 172.21.15.100:0/1955712146 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc750007070 con 0x7fc75c06ea50 2024-09-13T11:53:08.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc7437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:08.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7f688fa40640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:53:08.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc7437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:08.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc7437fe640 10 monclient: dump: 2024-09-13T11:53:08.203 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:08.203 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.204 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.204 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.204 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:08.204 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.204 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:08.204 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:08.204 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:08.204 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc76162a640 5 monclient: authenticate success, global_id 4320 2024-09-13T11:53:08.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc76162a640 10 monclient: _renew_subs 2024-09-13T11:53:08.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc76162a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:08.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc76162a640 1 -- 172.21.15.100:0/1955712146 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc75c0a10d0 con 0x7fc75c06ea50 2024-09-13T11:53:08.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc7597fa640 4 set_mon_vals no callback set 2024-09-13T11:53:08.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc76162a640 10 monclient: _renew_subs 2024-09-13T11:53:08.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc76162a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:08.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fc76162a640 1 -- 172.21.15.100:0/1955712146 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc75c120aa0 con 0x7fc75c06ea50 2024-09-13T11:53:08.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fb84db2b640 1 -- start start 2024-09-13T11:53:08.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fb84db2b640 10 monclient: init 2024-09-13T11:53:08.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7f688fa40640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f688806ea50 0x7f6888144180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.206 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7f688fa40640 10 monclient(hunting): picked mon.noname-b con 0x7f688806ea50 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:08.206 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fb84db2b640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:08.206 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fb84db2b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:53:08.206 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.083+0000 7fb84db2b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb84806ea50 0x7fb8481d1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.206 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb84db2b640 10 monclient(hunting): picked mon.noname-c con 0x7fb84806ea50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:08.206 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb84db2b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb84806f370 0x7fb8481d1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.206 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb84db2b640 10 monclient(hunting): picked mon.noname-b con 0x7fb84806f370 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:08.206 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb84db2b640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb84813c480 0x7fb848128c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.206 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb84db2b640 10 monclient(hunting): picked mon.noname-a con 0x7fb84813c480 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:08.206 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb84db2b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb84db2b640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fb84806fca0 con 0x7fb84813c480 2024-09-13T11:53:08.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb84db2b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb84db2b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb84806f7b0 con 0x7fb84806f370 2024-09-13T11:53:08.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb84db2b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb84db2b640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fb84812c1a0 con 0x7fb84806ea50 2024-09-13T11:53:08.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb84db2b640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:08.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb84db2b640 10 monclient(hunting): authenticate will time out at 768.677065s 2024-09-13T11:53:08.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb847fff640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb84813c480 0x7fb848128c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb847fff640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb84813c480 0x7fb848128c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49602/0 (socket says 172.21.15.100:49602) 2024-09-13T11:53:08.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb847fff640 1 -- 172.21.15.100:0/1165737907 learned_addr learned my addr 172.21.15.100:0/1165737907 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:08.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb847fff640 10 monclient(hunting): get_auth_request con 0x7fb84813c480 auth_method 0 2024-09-13T11:53:08.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb847fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb847fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb847fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb846ffd640 1 --2- 172.21.15.100:0/1165737907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb84806f370 0x7fb8481d1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb847fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:08.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb847fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:08.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb847fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:08.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb8477fe640 1 --2- 172.21.15.100:0/1165737907 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb84806ea50 0x7fb8481d1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb8477fe640 10 monclient(hunting): get_auth_request con 0x7fb84806ea50 auth_method 0 2024-09-13T11:53:08.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb8477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb8477fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb8477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7f688d7dc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f688806ea50 0x7f6888144180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.209 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb846ffd640 10 monclient(hunting): get_auth_request con 0x7fb84806f370 auth_method 0 2024-09-13T11:53:08.209 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb846ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.209 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb846ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.209 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb846ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.209 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb847fff640 10 monclient(hunting): handle_auth_done global_id 4323 payload 824 2024-09-13T11:53:08.209 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb847fff640 1 -- 172.21.15.100:0/1165737907 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb84806ea50 msgr2=0x7fb8481d1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.209 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb847fff640 1 --2- 172.21.15.100:0/1165737907 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb84806ea50 0x7fb8481d1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7f688d7dc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f688806ea50 0x7f6888144180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.100:50490/0 (socket says 172.21.15.100:50490) 2024-09-13T11:53:08.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb847fff640 1 -- 172.21.15.100:0/1165737907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb84806f370 msgr2=0x7fb8481d1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb847fff640 1 --2- 172.21.15.100:0/1165737907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb84806f370 0x7fb8481d1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb847fff640 10 monclient: _finish_hunting 0 2024-09-13T11:53:08.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb847fff640 1 monclient: found mon.noname-a 2024-09-13T11:53:08.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb847fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:08.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb847fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:08.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb847fff640 1 -- 172.21.15.100:0/1165737907 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8480a52e0 con 0x7fb84813c480 2024-09-13T11:53:08.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7f688d7dc640 1 -- 172.21.15.100:0/3647159772 learned_addr learned my addr 172.21.15.100:0/3647159772 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:08.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7f688fa40640 1 --2- 172.21.15.100:0/3647159772 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f688806f370 0x7f688813cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb8477fe640 1 --2- 172.21.15.100:0/1165737907 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb84806ea50 0x7fb8481d1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:53:08.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.084+0000 7fb847fff640 1 --2- 172.21.15.100:0/1165737907 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb84813c480 0x7fb848128c40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb84000cef0 tx=0x7fb84000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:08.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fb846ffd640 1 --2- 172.21.15.100:0/1165737907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb84806f370 0x7fb8481d1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:53:08.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fb83ffff640 1 -- 172.21.15.100:0/1165737907 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb84001a070 con 0x7fb84813c480 2024-09-13T11:53:08.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fb83ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:08.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fb83ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:08.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fb83ffff640 10 monclient: dump: 2024-09-13T11:53:08.212 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:08.212 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.212 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.212 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.212 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:08.212 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.212 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:08.212 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:08.212 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:08.212 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.213 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fb83ffff640 10 monclient: _finish_auth 0 2024-09-13T11:53:08.213 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fb83ffff640 10 monclient: _check_auth_tickets 2024-09-13T11:53:08.213 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fb83ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:08.213 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fb84db2b640 5 monclient: authenticate success, global_id 4323 2024-09-13T11:53:08.213 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fb84db2b640 10 monclient: _renew_subs 2024-09-13T11:53:08.213 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fb84db2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:08.213 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fb84db2b640 1 -- 172.21.15.100:0/1165737907 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb84813be00 con 0x7fb84813c480 2024-09-13T11:53:08.213 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fb84db2b640 10 monclient: _renew_subs 2024-09-13T11:53:08.213 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fb84db2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:08.213 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fb84db2b640 1 -- 172.21.15.100:0/1165737907 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb84813b480 con 0x7fb84813c480 2024-09-13T11:53:08.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688fa40640 10 monclient(hunting): picked mon.noname-a con 0x7f688806f370 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:08.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688fa40640 1 --2- 172.21.15.100:0/3647159772 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f688813c480 0x7f688813d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688fa40640 10 monclient(hunting): picked mon.noname-c con 0x7f688813c480 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:08.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688fa40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688fa40640 1 -- 172.21.15.100:0/3647159772 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f688806fca0 con 0x7f688806f370 2024-09-13T11:53:08.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688fa40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688fa40640 1 -- 172.21.15.100:0/3647159772 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f688806f7e0 con 0x7f688806ea50 2024-09-13T11:53:08.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688fa40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:08.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688fa40640 1 -- 172.21.15.100:0/3647159772 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f688813f030 con 0x7f688813c480 2024-09-13T11:53:08.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688fa40640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:08.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688cfdb640 1 --2- 172.21.15.100:0/3647159772 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f688806f370 0x7f688813cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688fa40640 10 monclient(hunting): authenticate will time out at 768.678071s 2024-09-13T11:53:08.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688dfdd640 1 --2- 172.21.15.100:0/3647159772 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f688813c480 0x7f688813d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688dfdd640 10 monclient(hunting): get_auth_request con 0x7f688813c480 auth_method 0 2024-09-13T11:53:08.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688dfdd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688dfdd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688dfdd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688cfdb640 10 monclient(hunting): get_auth_request con 0x7f688806f370 auth_method 0 2024-09-13T11:53:08.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688cfdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688cfdb640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688cfdb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fb83ffff640 1 -- 172.21.15.100:0/1165737907 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb84000d860 con 0x7fb84813c480 2024-09-13T11:53:08.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fb83ffff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:08.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fb83ffff640 1 -- 172.21.15.100:0/1165737907 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb84000dc50 con 0x7fb84813c480 2024-09-13T11:53:08.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fb83ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:08.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fb83ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:08.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fb83ffff640 10 monclient: dump: 2024-09-13T11:53:08.216 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:08.216 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.216 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.216 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.216 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:08.216 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.216 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:08.216 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:08.216 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:08.216 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fb845ffb640 4 set_mon_vals no callback set 2024-09-13T11:53:08.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fc76162a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:08.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688d7dc640 10 monclient(hunting): get_auth_request con 0x7f688806ea50 auth_method 0 2024-09-13T11:53:08.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688d7dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:08.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688d7dc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:08.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688d7dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:08.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688dfdd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:08.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688dfdd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:08.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688dfdd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:08.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688d7dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:08.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688d7dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:08.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688d7dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:08.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688dfdd640 10 monclient(hunting): handle_auth_done global_id 4292 payload 824 2024-09-13T11:53:08.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688dfdd640 1 -- 172.21.15.100:0/3647159772 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f688806ea50 msgr2=0x7f6888144180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688dfdd640 1 --2- 172.21.15.100:0/3647159772 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f688806ea50 0x7f6888144180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688dfdd640 1 -- 172.21.15.100:0/3647159772 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f688806f370 msgr2=0x7f688813cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688dfdd640 1 --2- 172.21.15.100:0/3647159772 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f688806f370 0x7f688813cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688dfdd640 10 monclient: _finish_hunting 0 2024-09-13T11:53:08.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688dfdd640 1 monclient: found mon.noname-c 2024-09-13T11:53:08.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688dfdd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:08.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688dfdd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:53:08.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7f688dfdd640 1 -- 172.21.15.100:0/3647159772 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68880a8df0 con 0x7f688813c480 2024-09-13T11:53:08.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fc76162a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:08.220 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fc76162a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:08.220 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.086+0000 7f688dfdd640 1 --2- 172.21.15.100:0/3647159772 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f688813c480 0x7f688813d4d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f68880a19f0 tx=0x7f6880004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:08.220 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fc76162a640 1 -- 172.21.15.100:0/1955712146 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc75c06ee30 con 0x7fc75c06ea50 2024-09-13T11:53:08.220 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.085+0000 7fc7437fe640 1 -- 172.21.15.100:0/1955712146 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fc75001a070 con 0x7fc75c06ea50 2024-09-13T11:53:08.221 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.086+0000 7fc7437fe640 1 --2- 172.21.15.100:0/1955712146 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc74803dff0 0x7fc7480404b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.221 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.086+0000 7fb83ffff640 1 -- 172.21.15.100:0/1165737907 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fb84001d070 con 0x7fb84813c480 2024-09-13T11:53:08.221 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.086+0000 7fb83ffff640 1 --2- 172.21.15.100:0/1165737907 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fb83003dfa0 0x7fb830040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.221 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.086+0000 7fb83ffff640 1 -- 172.21.15.100:0/1165737907 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7fb840052e50 con 0x7fb84813c480 2024-09-13T11:53:08.222 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.086+0000 7fc7437fe640 1 -- 172.21.15.100:0/1955712146 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7fc750051950 con 0x7fc75c06ea50 2024-09-13T11:53:08.222 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7fc75a7fc640 1 --2- 172.21.15.100:0/1955712146 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc74803dff0 0x7fc7480404b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.222 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7fc75a7fc640 10 monclient: get_auth_request con 0x7fc74803dff0 auth_method 0 2024-09-13T11:53:08.223 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7fb8477fe640 1 --2- 172.21.15.100:0/1165737907 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fb83003dfa0 0x7fb830040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.223 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7fb8477fe640 10 monclient: get_auth_request con 0x7fb83003dfa0 auth_method 0 2024-09-13T11:53:08.223 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7fb84db2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:08.223 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7fb84db2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:08.223 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7fb84db2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:08.223 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7fb84db2b640 1 -- 172.21.15.100:0/1165737907 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb824005080 con 0x7fb84813c480 2024-09-13T11:53:08.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7f687dffb640 1 -- 172.21.15.100:0/3647159772 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6880041020 con 0x7f688813c480 2024-09-13T11:53:08.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7f687dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:08.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7f687dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:08.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7f687dffb640 10 monclient: dump: 2024-09-13T11:53:08.224 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:08.224 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.224 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.224 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.224 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:08.225 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.225 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:08.225 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:08.225 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:08.225 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7f687dffb640 10 monclient: _finish_auth 0 2024-09-13T11:53:08.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7f687dffb640 10 monclient: _check_auth_tickets 2024-09-13T11:53:08.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7f687dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:08.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7f687dffb640 1 -- 172.21.15.100:0/3647159772 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68800089b0 con 0x7f688813c480 2024-09-13T11:53:08.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7f688fa40640 5 monclient: authenticate success, global_id 4292 2024-09-13T11:53:08.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7f688fa40640 10 monclient: _renew_subs 2024-09-13T11:53:08.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7f688fa40640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:08.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7f688fa40640 1 -- 172.21.15.100:0/3647159772 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6888140720 con 0x7f688813c480 2024-09-13T11:53:08.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7f687dffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:08.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7f687dffb640 1 -- 172.21.15.100:0/3647159772 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6880036040 con 0x7f688813c480 2024-09-13T11:53:08.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7f687dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:08.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7f687dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:08.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7f3ef9ffb640 1 -- 172.21.15.100:0/1462170745 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f3efc010300 con 0x7f3f04136b00 2024-09-13T11:53:08.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7f3ef9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:08.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7f687dffb640 10 monclient: dump: 2024-09-13T11:53:08.226 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:08.227 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:08.227 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.227 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:08.227 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:08.227 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:08.227 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:08.227 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:08.227 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:08.227 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:08.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7f3ef9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:08.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.088+0000 7fc75a7fc640 1 --2- 172.21.15.100:0/1955712146 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc74803dff0 0x7fc7480404b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc74c0097d0 tx=0x7fc74c02e000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:08.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7f688fa40640 10 monclient: _renew_subs 2024-09-13T11:53:08.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.088+0000 7f688fa40640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:08.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.088+0000 7f688fa40640 1 -- 172.21.15.100:0/3647159772 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f688813f200 con 0x7f688813c480 2024-09-13T11:53:08.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.088+0000 7fb8477fe640 1 --2- 172.21.15.100:0/1165737907 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fb83003dfa0 0x7fb830040460 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb838034000 tx=0x7fb83802e000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:08.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.087+0000 7f687f7fe640 4 set_mon_vals no callback set 2024-09-13T11:53:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.088+0000 7f687dffb640 1 -- 172.21.15.100:0/3647159772 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f6880044020 con 0x7f688813c480 2024-09-13T11:53:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.089+0000 7f687dffb640 1 --2- 172.21.15.100:0/3647159772 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f686c03dfa0 0x7f686c040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.089+0000 7f687dffb640 1 -- 172.21.15.100:0/3647159772 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7f6880073570 con 0x7f688813c480 2024-09-13T11:53:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.089+0000 7f688d7dc640 1 --2- 172.21.15.100:0/3647159772 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f686c03dfa0 0x7f686c040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.089+0000 7f688fa40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.089+0000 7f688fa40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.089+0000 7f688fa40640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.089+0000 7f688fa40640 1 -- 172.21.15.100:0/3647159772 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68880a5a60 con 0x7f688813c480 2024-09-13T11:53:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.089+0000 7f688d7dc640 10 monclient: get_auth_request con 0x7f686c03dfa0 auth_method 0 2024-09-13T11:53:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.090+0000 7f688d7dc640 1 --2- 172.21.15.100:0/3647159772 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f686c03dfa0 0x7f686c040460 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6884009b60 tx=0x7f6884009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.094+0000 7f687dffb640 1 -- 172.21.15.100:0/3647159772 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f688004b030 con 0x7f688813c480 2024-09-13T11:53:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.094+0000 7f687dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.094+0000 7f687dffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:08.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.095+0000 7fc7437fe640 1 -- 172.21.15.100:0/1955712146 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc75c0a10d0 con 0x7fc75c06ea50 2024-09-13T11:53:08.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.095+0000 7fc7437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:08.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.095+0000 7fc7437fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:08.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.104+0000 7fb83ffff640 1 -- 172.21.15.100:0/1165737907 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fb84001ec50 con 0x7fb84813c480 2024-09-13T11:53:08.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.104+0000 7fb83ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:08.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.104+0000 7fb83ffff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:08.281 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.278+0000 7fc76162a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-13T11:53:08.282 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.278+0000 7fc76162a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-13T11:53:08.282 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.278+0000 7fc76162a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:08.282 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.278+0000 7fc76162a640 1 -- 172.21.15.100:0/1955712146 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fc75c1385a0 con 0x7fc75c06ea50 2024-09-13T11:53:08.282 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.279+0000 7fc7437fe640 1 -- 172.21.15.100:0/1955712146 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc75c06fa80 con 0x7fc75c06ea50 2024-09-13T11:53:08.282 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.279+0000 7fc7437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-13T11:53:08.283 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.279+0000 7fc7437fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:08.283 INFO:teuthology.orchestra.run.smithi100.stdout:47244640260 2024-09-13T11:53:08.284 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.281+0000 7fc76162a640 1 -- 172.21.15.100:0/1955712146 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc74803dff0 msgr2=0x7fc7480404b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.284 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.281+0000 7fc76162a640 1 --2- 172.21.15.100:0/1955712146 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc74803dff0 0x7fc7480404b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc74c0097d0 tx=0x7fc74c02e000 comp rx=0 tx=0).stop 2024-09-13T11:53:08.284 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.281+0000 7fc76162a640 10 monclient: shutdown 2024-09-13T11:53:08.284 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.281+0000 7fc76162a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:08.284 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.281+0000 7fc76162a640 1 -- 172.21.15.100:0/1955712146 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc75c06ea50 msgr2=0x7fc75c1376e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.284 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.281+0000 7fc76162a640 1 --2- 172.21.15.100:0/1955712146 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc75c06ea50 0x7fc75c1376e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc7500013f0 tx=0x7fc75000cc00 comp rx=0 tx=0).stop 2024-09-13T11:53:08.284 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.281+0000 7fc76162a640 1 -- 172.21.15.100:0/1955712146 shutdown_connections 2024-09-13T11:53:08.284 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.281+0000 7fc76162a640 1 --2- 172.21.15.100:0/1955712146 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc74803dff0 0x7fc7480404b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.284 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.281+0000 7fc76162a640 1 --2- 172.21.15.100:0/1955712146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc75c13c9f0 0x7fc75c09f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.284 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.281+0000 7fc76162a640 1 --2- 172.21.15.100:0/1955712146 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc75c0a2860 0x7fc75c09ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.285 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.281+0000 7fc76162a640 1 --2- 172.21.15.100:0/1955712146 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc75c06ea50 0x7fc75c1376e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.285 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.281+0000 7fc76162a640 1 -- 172.21.15.100:0/1955712146 >> 172.21.15.100:0/1955712146 conn(0x7fc75c0a68e0 msgr2=0x7fc75c0a91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:08.285 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.281+0000 7fc76162a640 1 -- 172.21.15.100:0/1955712146 shutdown_connections 2024-09-13T11:53:08.285 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.281+0000 7fc76162a640 1 -- 172.21.15.100:0/1955712146 wait complete. 2024-09-13T11:53:08.286 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.281+0000 7f688fa40640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-13T11:53:08.286 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.281+0000 7f688fa40640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-13T11:53:08.286 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.281+0000 7f688fa40640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:08.286 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.281+0000 7f688fa40640 1 -- 172.21.15.100:0/3647159772 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f6888069100 con 0x7f688813c480 2024-09-13T11:53:08.286 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.282+0000 7f687dffb640 1 -- 172.21.15.100:0/3647159772 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f68880a8df0 con 0x7f688813c480 2024-09-13T11:53:08.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.282+0000 7f687dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-13T11:53:08.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.282+0000 7f687dffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:08.287 INFO:teuthology.orchestra.run.smithi100.stdout:51539607555 2024-09-13T11:53:08.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.284+0000 7f688fa40640 1 -- 172.21.15.100:0/3647159772 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f686c03dfa0 msgr2=0x7f686c040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.284+0000 7f688fa40640 1 --2- 172.21.15.100:0/3647159772 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f686c03dfa0 0x7f686c040460 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6884009b60 tx=0x7f6884009190 comp rx=0 tx=0).stop 2024-09-13T11:53:08.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.284+0000 7f688fa40640 10 monclient: shutdown 2024-09-13T11:53:08.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.284+0000 7f688fa40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:08.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.284+0000 7f688fa40640 1 -- 172.21.15.100:0/3647159772 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f688813c480 msgr2=0x7f688813d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.284+0000 7f688fa40640 1 --2- 172.21.15.100:0/3647159772 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f688813c480 0x7f688813d4d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f68880a19f0 tx=0x7f6880004a40 comp rx=0 tx=0).stop 2024-09-13T11:53:08.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.284+0000 7f688fa40640 1 -- 172.21.15.100:0/3647159772 shutdown_connections 2024-09-13T11:53:08.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.284+0000 7f688fa40640 1 --2- 172.21.15.100:0/3647159772 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f686c03dfa0 0x7f686c040460 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.284+0000 7f688fa40640 1 --2- 172.21.15.100:0/3647159772 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f688806f370 0x7f688813cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.284+0000 7f688fa40640 1 --2- 172.21.15.100:0/3647159772 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f688813c480 0x7f688813d4d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.284+0000 7f688fa40640 1 --2- 172.21.15.100:0/3647159772 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f688806ea50 0x7f6888144180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.284+0000 7f688fa40640 1 -- 172.21.15.100:0/3647159772 >> 172.21.15.100:0/3647159772 conn(0x7f68880a68e0 msgr2=0x7f68880a8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:08.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.285+0000 7f688fa40640 1 -- 172.21.15.100:0/3647159772 shutdown_connections 2024-09-13T11:53:08.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.285+0000 7f688fa40640 1 -- 172.21.15.100:0/3647159772 wait complete. 2024-09-13T11:53:08.294 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.2 2024-09-13T11:53:08.297 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607555 for osd.3 2024-09-13T11:53:08.388 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.387+0000 7f9659649640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-13T11:53:08.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.387+0000 7f9659649640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-13T11:53:08.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.387+0000 7f9659649640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:08.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.387+0000 7f9659649640 1 -- 172.21.15.100:0/1271138643 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f96540a0220 con 0x7f965406ea50 2024-09-13T11:53:08.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.387+0000 7f964b7fe640 1 -- 172.21.15.100:0/1271138643 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f96540a0220 con 0x7f965406ea50 2024-09-13T11:53:08.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.387+0000 7f964b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-13T11:53:08.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.387+0000 7f964b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:08.389 INFO:teuthology.orchestra.run.smithi100.stdout:47244640261 2024-09-13T11:53:08.391 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.389+0000 7f9659649640 1 -- 172.21.15.100:0/1271138643 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f963403dfa0 msgr2=0x7f9634040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.391 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.389+0000 7f9659649640 1 --2- 172.21.15.100:0/1271138643 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f963403dfa0 0x7f9634040460 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f963c034000 tx=0x7f963c02e000 comp rx=0 tx=0).stop 2024-09-13T11:53:08.391 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.389+0000 7f9659649640 10 monclient: shutdown 2024-09-13T11:53:08.391 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.389+0000 7f9659649640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:08.391 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.389+0000 7f9659649640 1 -- 172.21.15.100:0/1271138643 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f965406ea50 msgr2=0x7f96541cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.391 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.389+0000 7f9659649640 1 --2- 172.21.15.100:0/1271138643 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f965406ea50 0x7f96541cc730 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f964400c440 tx=0x7f964400c900 comp rx=0 tx=0).stop 2024-09-13T11:53:08.391 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.389+0000 7f9659649640 1 -- 172.21.15.100:0/1271138643 shutdown_connections 2024-09-13T11:53:08.391 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.389+0000 7f9659649640 1 --2- 172.21.15.100:0/1271138643 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f963403dfa0 0x7f9634040460 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.392 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.389+0000 7f9659649640 1 --2- 172.21.15.100:0/1271138643 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96540a2860 0x7f9654137880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.392 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.389+0000 7f9659649640 1 --2- 172.21.15.100:0/1271138643 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f965413c9f0 0x7f965409dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.392 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.389+0000 7f9659649640 1 --2- 172.21.15.100:0/1271138643 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f965406ea50 0x7f96541cc730 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.392 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.389+0000 7f9659649640 1 -- 172.21.15.100:0/1271138643 >> 172.21.15.100:0/1271138643 conn(0x7f96540a68e0 msgr2=0x7f965413bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:08.392 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.389+0000 7f9659649640 1 -- 172.21.15.100:0/1271138643 shutdown_connections 2024-09-13T11:53:08.392 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.389+0000 7f9659649640 1 -- 172.21.15.100:0/1271138643 wait complete. 2024-09-13T11:53:08.393 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.390+0000 7f57e06b6640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-13T11:53:08.393 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.390+0000 7f57e06b6640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-13T11:53:08.393 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.390+0000 7f57e06b6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:53:08.393 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.390+0000 7f57e06b6640 1 -- 172.21.15.100:0/477806134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f57d80008d0 con 0x7f57d806ea50 2024-09-13T11:53:08.393 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.390+0000 7f57ceffd640 1 -- 172.21.15.100:0/477806134 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f57d403c380 con 0x7f57d806ea50 2024-09-13T11:53:08.393 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.390+0000 7f57ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-13T11:53:08.393 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.390+0000 7f57ceffd640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:08.394 INFO:teuthology.orchestra.run.smithi100.stdout:51539607556 2024-09-13T11:53:08.394 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.392+0000 7f57e06b6640 1 -- 172.21.15.100:0/477806134 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f57bc03dfa0 msgr2=0x7f57bc040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.392+0000 7f57e06b6640 1 --2- 172.21.15.100:0/477806134 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f57bc03dfa0 0x7f57bc040460 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f57d8137590 tx=0x7f57c8008040 comp rx=0 tx=0).stop 2024-09-13T11:53:08.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.392+0000 7f57e06b6640 10 monclient: shutdown 2024-09-13T11:53:08.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.392+0000 7f57e06b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:08.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.392+0000 7f57e06b6640 1 -- 172.21.15.100:0/477806134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f57d806ea50 msgr2=0x7f57d8123d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.392+0000 7f57e06b6640 1 --2- 172.21.15.100:0/477806134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f57d806ea50 0x7f57d8123d40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f57d402a720 tx=0x7f57d40046c0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.393+0000 7f57e06b6640 1 -- 172.21.15.100:0/477806134 shutdown_connections 2024-09-13T11:53:08.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.393+0000 7f57e06b6640 1 --2- 172.21.15.100:0/477806134 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f57bc03dfa0 0x7f57bc040460 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.393+0000 7f57e06b6640 1 --2- 172.21.15.100:0/477806134 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f57d806ee30 0x7f57d8124280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.393+0000 7f57e06b6640 1 --2- 172.21.15.100:0/477806134 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57d806f790 0x7f57d81247c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.393+0000 7f57e06b6640 1 --2- 172.21.15.100:0/477806134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f57d806ea50 0x7f57d8123d40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.393+0000 7f57e06b6640 1 -- 172.21.15.100:0/477806134 >> 172.21.15.100:0/477806134 conn(0x7f57d80a68e0 msgr2=0x7f57d809fb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:08.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.393+0000 7f57e06b6640 1 -- 172.21.15.100:0/477806134 shutdown_connections 2024-09-13T11:53:08.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.393+0000 7f57e06b6640 1 -- 172.21.15.100:0/477806134 wait complete. 2024-09-13T11:53:08.403 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640261 for osd.5 2024-09-13T11:53:08.406 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607556 for osd.0 2024-09-13T11:53:08.442 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.439+0000 7f3f0b857640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-13T11:53:08.442 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.440+0000 7f3f0b857640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-13T11:53:08.442 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.440+0000 7f3f0b857640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:08.442 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.440+0000 7f3f0b857640 1 -- 172.21.15.100:0/1462170745 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f3f0409f9d0 con 0x7f3f04136b00 2024-09-13T11:53:08.443 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.440+0000 7f3ef9ffb640 1 -- 172.21.15.100:0/1462170745 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3efc007980 con 0x7f3f04136b00 2024-09-13T11:53:08.443 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.440+0000 7f3ef9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-13T11:53:08.443 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.440+0000 7f3ef9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:08.443 INFO:teuthology.orchestra.run.smithi100.stdout:47244640261 2024-09-13T11:53:08.444 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.442+0000 7f3f0b857640 1 -- 172.21.15.100:0/1462170745 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f3ee803dfa0 msgr2=0x7f3ee8040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.444 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.442+0000 7f3f0b857640 1 --2- 172.21.15.100:0/1462170745 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f3ee803dfa0 0x7f3ee8040460 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3f041377e0 tx=0x7f3f00006bd0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.444 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.442+0000 7f3f0b857640 10 monclient: shutdown 2024-09-13T11:53:08.444 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.442+0000 7f3f0b857640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:08.444 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.442+0000 7f3f0b857640 1 -- 172.21.15.100:0/1462170745 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3f04136b00 msgr2=0x7f3f0409cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.444 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.442+0000 7f3f0b857640 1 --2- 172.21.15.100:0/1462170745 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3f04136b00 0x7f3f0409cf90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3efc00c7f0 tx=0x7f3efc00ccb0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.444 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.442+0000 7f3f0b857640 1 -- 172.21.15.100:0/1462170745 shutdown_connections 2024-09-13T11:53:08.444 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.442+0000 7f3f0b857640 1 --2- 172.21.15.100:0/1462170745 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f3ee803dfa0 0x7f3ee8040460 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.444 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.442+0000 7f3f0b857640 1 --2- 172.21.15.100:0/1462170745 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3f0413ce90 0x7f3f0409ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.442+0000 7f3f0b857640 1 --2- 172.21.15.100:0/1462170745 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3f04136b00 0x7f3f0409cf90 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.442+0000 7f3f0b857640 1 --2- 172.21.15.100:0/1462170745 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f04137a60 0x7f3f041365c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.442+0000 7f3f0b857640 1 -- 172.21.15.100:0/1462170745 >> 172.21.15.100:0/1462170745 conn(0x7f3f040aa580 msgr2=0x7f3f040a3700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:08.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.442+0000 7f3f0b857640 1 -- 172.21.15.100:0/1462170745 shutdown_connections 2024-09-13T11:53:08.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.442+0000 7f3f0b857640 1 -- 172.21.15.100:0/1462170745 wait complete. 2024-09-13T11:53:08.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.447+0000 7fb84db2b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-13T11:53:08.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.447+0000 7fb84db2b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-13T11:53:08.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.447+0000 7fb84db2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:08.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.447+0000 7fb84db2b640 1 -- 172.21.15.100:0/1165737907 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fb824005640 con 0x7fb84813c480 2024-09-13T11:53:08.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.447+0000 7fb83ffff640 1 -- 172.21.15.100:0/1165737907 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb8400143a0 con 0x7fb84813c480 2024-09-13T11:53:08.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.447+0000 7fb83ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-13T11:53:08.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.447+0000 7fb83ffff640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:08.449 INFO:teuthology.orchestra.run.smithi100.stdout:51539607555 2024-09-13T11:53:08.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.449+0000 7fb84db2b640 1 -- 172.21.15.100:0/1165737907 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fb83003dfa0 msgr2=0x7fb830040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.449+0000 7fb84db2b640 1 --2- 172.21.15.100:0/1165737907 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fb83003dfa0 0x7fb830040460 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb838034000 tx=0x7fb83802e000 comp rx=0 tx=0).stop 2024-09-13T11:53:08.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.449+0000 7fb84db2b640 10 monclient: shutdown 2024-09-13T11:53:08.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.449+0000 7fb84db2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:08.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.449+0000 7fb84db2b640 1 -- 172.21.15.100:0/1165737907 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb84813c480 msgr2=0x7fb848128c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:08.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.449+0000 7fb84db2b640 1 --2- 172.21.15.100:0/1165737907 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb84813c480 0x7fb848128c40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb84000cef0 tx=0x7fb84000a710 comp rx=0 tx=0).stop 2024-09-13T11:53:08.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.449+0000 7fb84db2b640 1 -- 172.21.15.100:0/1165737907 shutdown_connections 2024-09-13T11:53:08.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.449+0000 7fb84db2b640 1 --2- 172.21.15.100:0/1165737907 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fb83003dfa0 0x7fb830040460 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.449+0000 7fb84db2b640 1 --2- 172.21.15.100:0/1165737907 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb84813c480 0x7fb848128c40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.449+0000 7fb84db2b640 1 --2- 172.21.15.100:0/1165737907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb84806f370 0x7fb8481d1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.449+0000 7fb84db2b640 1 --2- 172.21.15.100:0/1165737907 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb84806ea50 0x7fb8481d1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:08.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.449+0000 7fb84db2b640 1 -- 172.21.15.100:0/1165737907 >> 172.21.15.100:0/1165737907 conn(0x7fb8480a68e0 msgr2=0x7fb848072e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:08.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.449+0000 7fb84db2b640 1 -- 172.21.15.100:0/1165737907 shutdown_connections 2024-09-13T11:53:08.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:08.449+0000 7fb84db2b640 1 -- 172.21.15.100:0/1165737907 wait complete. 2024-09-13T11:53:08.454 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640261 for osd.1 2024-09-13T11:53:08.461 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607555 for osd.4 2024-09-13T11:53:09.295 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T11:53:09.298 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T11:53:09.403 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T11:53:09.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.405+0000 7f1f8ea3c640 10 monclient: get_monmap_and_config 2024-09-13T11:53:09.407 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.405+0000 7f1f8ea3c640 10 monclient: build_initial_monmap 2024-09-13T11:53:09.407 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.405+0000 7f1f8ea3c640 10 monclient: monmap: 2024-09-13T11:53:09.407 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:09.407 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.407 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:09.406263+0000 2024-09-13T11:53:09.407 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:09.406263+0000 2024-09-13T11:53:09.407 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:09.407 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.407 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:09.407 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:09.407 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:09.407 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:09.408 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:09.408 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.408 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T11:53:09.409 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f1f8ea3c640 1 Processor -- start 2024-09-13T11:53:09.409 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f1f8ea3c640 1 -- start start 2024-09-13T11:53:09.409 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f1f8ea3c640 10 monclient: init 2024-09-13T11:53:09.409 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f0175639640 10 monclient: get_monmap_and_config 2024-09-13T11:53:09.409 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f0175639640 10 monclient: build_initial_monmap 2024-09-13T11:53:09.409 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f0175639640 10 monclient: monmap: 2024-09-13T11:53:09.409 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:09.410 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.410 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:09.407219+0000 2024-09-13T11:53:09.410 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:09.407219+0000 2024-09-13T11:53:09.410 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:09.410 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.410 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:09.410 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:09.410 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:09.410 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:09.410 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:09.410 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.411 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f1f8ea3c640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:09.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f1f8ea3c640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-13T11:53:09.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f1f8ea3c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1f88137a60 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:09.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f1f8ea3c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1f880a2980 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:09.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f1f8ea3c640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f8813e070 0x7f1f88140460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f1f8ea3c640 10 monclient(hunting): picked mon.noname-b con 0x7f1f8813e070 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:09.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f1f8ea3c640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f88138550 con 0x7f1f880a2980 2024-09-13T11:53:09.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f1f8ea3c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f880a5be0 con 0x7f1f88137a60 2024-09-13T11:53:09.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f1f8ea3c640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f1f8ea3c640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f1f880a4410 con 0x7f1f8813e070 2024-09-13T11:53:09.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f1f8ea3c640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:09.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f1f8ea3c640 10 monclient(hunting): authenticate will time out at 769.999802s 2024-09-13T11:53:09.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f1f8cfd9640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f8813e070 0x7f1f88140460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f1f877fe640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f1f880a2980 0x7f1f8813c950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:58016/0 (socket says 172.21.15.100:58016) 2024-09-13T11:53:09.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f1f877fe640 1 -- 172.21.15.100:0/3524252059 learned_addr learned my addr 172.21.15.100:0/3524252059 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:53:09.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f1f8cfd9640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f8813e070 0x7f1f88140460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47432/0 (socket says 172.21.15.100:47432) 2024-09-13T11:53:09.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f1f8cfd9640 10 monclient(hunting): get_auth_request con 0x7f1f8813e070 auth_method 0 2024-09-13T11:53:09.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f1f8cfd9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f1f8cfd9640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.406+0000 7f1f8cfd9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f1f8cfd9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f1f8cfd9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f0175639640 1 Processor -- start 2024-09-13T11:53:09.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f1f8cfd9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f0175639640 1 -- start start 2024-09-13T11:53:09.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f0175639640 10 monclient: init 2024-09-13T11:53:09.415 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f1f8cfd9640 10 monclient(hunting): handle_auth_done global_id 4295 payload 274 2024-09-13T11:53:09.415 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f1f86ffd640 1 -- 172.21.15.100:0/3524252059 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 300423795 0 0) 0x7f1f88138550 con 0x7f1f880a2980 2024-09-13T11:53:09.415 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f1f8cfd9640 1 -- 172.21.15.100:0/3524252059 >> v1:172.21.15.122:6789/0 conn(0x7f1f88137a60 legacy=0x7f1f88137e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.415 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f1f8cfd9640 1 -- 172.21.15.100:0/3524252059 >> v1:172.21.15.100:6789/0 conn(0x7f1f880a2980 legacy=0x7f1f8813c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.415 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f1f8cfd9640 10 monclient: _finish_hunting 0 2024-09-13T11:53:09.415 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f1f8cfd9640 1 monclient: found mon.noname-b 2024-09-13T11:53:09.415 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f1f8cfd9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:09.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f1f8cfd9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:53:09.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f1f8cfd9640 1 -- 172.21.15.100:0/3524252059 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f880a76b0 con 0x7f1f8813e070 2024-09-13T11:53:09.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f1f86ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:53:09.418 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f0175639640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:09.418 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f0175639640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-13T11:53:09.418 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f0175639640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0170137880 0x7f0170137c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.418 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f0175639640 10 monclient(hunting): picked mon.noname-b con 0x7f0170137880 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:09.418 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f0175639640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f01700a26e0 0x7f01700a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.418 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f0175639640 10 monclient(hunting): picked mon.noname-c con 0x7f01700a26e0 addr v2:172.21.15.122:3300/0 2024-09-13T11:53:09.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f0175639640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f01700a3040 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:09.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f0175639640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01700a34f0 con 0x7f01700a3040 2024-09-13T11:53:09.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f0175639640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f0175639640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f0170076630 con 0x7f01700a26e0 2024-09-13T11:53:09.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f0175639640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f0175639640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f01700761d0 con 0x7f0170137880 2024-09-13T11:53:09.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f0175639640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:09.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f0175639640 10 monclient(hunting): authenticate will time out at 770.000802s 2024-09-13T11:53:09.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f016effd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0170137880 0x7f0170137c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f016effd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0170137880 0x7f0170137c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47436/0 (socket says 172.21.15.100:47436) 2024-09-13T11:53:09.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f016effd640 1 -- 172.21.15.100:0/1472244147 learned_addr learned my addr 172.21.15.100:0/1472244147 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:09.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f016effd640 10 monclient(hunting): get_auth_request con 0x7f0170137880 auth_method 0 2024-09-13T11:53:09.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f016effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f016effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.420 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f016effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.420 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f016e7fc640 1 --2- 172.21.15.100:0/1472244147 >> v2:172.21.15.122:3300/0 conn(0x7f01700a26e0 0x7f01700a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.420 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f016e7fc640 10 monclient(hunting): get_auth_request con 0x7f01700a26e0 auth_method 0 2024-09-13T11:53:09.420 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f016e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.420 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f016e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.420 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.407+0000 7f016e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.420 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f016effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.420 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f016effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.420 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f016effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.420 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f016e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.420 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f016e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.420 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f016e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.420 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f016effd640 10 monclient(hunting): handle_auth_done global_id 4298 payload 274 2024-09-13T11:53:09.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f016effd640 1 -- 172.21.15.100:0/1472244147 >> v2:172.21.15.122:3300/0 conn(0x7f01700a26e0 msgr2=0x7f01700a2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:09.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f016effd640 1 --2- 172.21.15.100:0/1472244147 >> v2:172.21.15.122:3300/0 conn(0x7f01700a26e0 0x7f01700a2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f016effd640 1 -- 172.21.15.100:0/1472244147 >> v1:172.21.15.122:6789/0 conn(0x7f01700a3040 legacy=0x7f017013e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f016effd640 10 monclient: _finish_hunting 0 2024-09-13T11:53:09.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f016effd640 1 monclient: found mon.noname-b 2024-09-13T11:53:09.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f016effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:09.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f016effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:53:09.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f016effd640 1 -- 172.21.15.100:0/1472244147 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f017009c5c0 con 0x7f0170137880 2024-09-13T11:53:09.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f1f8cfd9640 1 --2- 172.21.15.100:0/3524252059 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f8813e070 0x7f1f88140460 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1f80009dc0 tx=0x7f1f8002a5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5378a9a4dfeede server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:09.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f1f86ffd640 1 -- 172.21.15.100:0/3524252059 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f80007740 con 0x7f1f8813e070 2024-09-13T11:53:09.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f1f86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:09.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f1f86ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:09.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f1f86ffd640 10 monclient: dump: 2024-09-13T11:53:09.422 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:09.422 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.422 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.423 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.423 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:09.423 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.423 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:09.423 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:09.423 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:09.423 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.423 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f1f86ffd640 10 monclient: _finish_auth 0 2024-09-13T11:53:09.423 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f1f86ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:53:09.423 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f1f86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:09.423 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f1f86ffd640 1 -- 172.21.15.100:0/3524252059 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f800078e0 con 0x7f1f8813e070 2024-09-13T11:53:09.423 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f1f8ea3c640 5 monclient: authenticate success, global_id 4295 2024-09-13T11:53:09.423 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f1f8ea3c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:09.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f1f86ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:09.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f1f8ea3c640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:09.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f1f86ffd640 1 -- 172.21.15.100:0/3524252059 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f80007cb0 con 0x7f1f8813e070 2024-09-13T11:53:09.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f1f8ea3c640 4 set_mon_vals no callback set 2024-09-13T11:53:09.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f1f86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:09.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f1f86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:09.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.408+0000 7f1f86ffd640 10 monclient: dump: 2024-09-13T11:53:09.424 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:09.424 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.424 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.424 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.424 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:09.424 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.425 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:09.425 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:09.425 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:09.425 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f016effd640 1 --2- 172.21.15.100:0/1472244147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0170137880 0x7f0170137c60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f01600098a0 tx=0x7f016002a200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe26759aa55026a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:09.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f016dffb640 1 -- 172.21.15.100:0/1472244147 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0160037070 con 0x7f0170137880 2024-09-13T11:53:09.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f016dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:09.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f016dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:09.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f016dffb640 10 monclient: dump: 2024-09-13T11:53:09.427 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:09.427 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.427 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.427 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.427 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:09.428 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.428 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:09.428 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:09.428 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:09.428 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f016dffb640 10 monclient: _finish_auth 0 2024-09-13T11:53:09.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f016dffb640 10 monclient: _check_auth_tickets 2024-09-13T11:53:09.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f016dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:09.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f0175639640 5 monclient: authenticate success, global_id 4298 2024-09-13T11:53:09.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f016dffb640 1 -- 172.21.15.100:0/1472244147 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0160032440 con 0x7f0170137880 2024-09-13T11:53:09.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f016dffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:09.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f0175639640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:09.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f0175639640 4 set_mon_vals no callback set 2024-09-13T11:53:09.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f016dffb640 1 -- 172.21.15.100:0/1472244147 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0160032810 con 0x7f0170137880 2024-09-13T11:53:09.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f016dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:09.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f016dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:09.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f016dffb640 10 monclient: dump: 2024-09-13T11:53:09.429 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:09.429 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.429 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.429 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.429 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:09.429 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.429 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:09.430 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:09.430 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:09.430 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.430 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f1f8ea3c640 10 monclient: shutdown 2024-09-13T11:53:09.430 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f1f8ea3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:09.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f1f8ea3c640 1 -- 172.21.15.100:0/3524252059 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f8813e070 msgr2=0x7f1f88140460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f1f8ea3c640 1 --2- 172.21.15.100:0/3524252059 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f8813e070 0x7f1f88140460 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1f80009dc0 tx=0x7f1f8002a5b0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f1f8ea3c640 1 -- 172.21.15.100:0/3524252059 shutdown_connections 2024-09-13T11:53:09.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f1f8ea3c640 1 --2- 172.21.15.100:0/3524252059 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f8813e070 0x7f1f88140460 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f1f8ea3c640 1 -- 172.21.15.100:0/3524252059 >> 172.21.15.100:0/3524252059 conn(0x7f1f880aa580 msgr2=0x7f1f880a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:09.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f1f8ea3c640 1 -- 172.21.15.100:0/3524252059 shutdown_connections 2024-09-13T11:53:09.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f1f8ea3c640 1 -- 172.21.15.100:0/3524252059 wait complete. 2024-09-13T11:53:09.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f0175639640 10 monclient: shutdown 2024-09-13T11:53:09.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f0175639640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:09.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f0175639640 1 -- 172.21.15.100:0/1472244147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0170137880 msgr2=0x7f0170137c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.409+0000 7f0175639640 1 --2- 172.21.15.100:0/1472244147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0170137880 0x7f0170137c60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f01600098a0 tx=0x7f016002a200 comp rx=0 tx=0).stop 2024-09-13T11:53:09.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.410+0000 7f0175639640 1 -- 172.21.15.100:0/1472244147 shutdown_connections 2024-09-13T11:53:09.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.410+0000 7f0175639640 1 --2- 172.21.15.100:0/1472244147 >> v2:172.21.15.122:3300/0 conn(0x7f01700a26e0 0x7f01700a2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.410+0000 7f0175639640 1 --2- 172.21.15.100:0/1472244147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0170137880 0x7f0170137c60 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.410+0000 7f0175639640 1 -- 172.21.15.100:0/1472244147 >> 172.21.15.100:0/1472244147 conn(0x7f01700a37d0 msgr2=0x7f01700a3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:09.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.410+0000 7f0175639640 1 -- 172.21.15.100:0/1472244147 shutdown_connections 2024-09-13T11:53:09.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.410+0000 7f0175639640 1 -- 172.21.15.100:0/1472244147 wait complete. 2024-09-13T11:53:09.433 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.410+0000 7f1f8ea3c640 10 monclient: build_initial_monmap 2024-09-13T11:53:09.433 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.410+0000 7f1f8ea3c640 10 monclient: monmap: 2024-09-13T11:53:09.433 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:09.434 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:09.434 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:09.434 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:09.434 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:09.434 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.434 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:09.434 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:09.434 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:09.434 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.410+0000 7f1f8ea3c640 1 Processor -- start 2024-09-13T11:53:09.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.410+0000 7f0175639640 10 monclient: build_initial_monmap 2024-09-13T11:53:09.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.410+0000 7f0175639640 10 monclient: monmap: 2024-09-13T11:53:09.435 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:09.435 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:09.435 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:09.435 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:09.435 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:09.435 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.435 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:09.435 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:09.435 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:09.435 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.410+0000 7f1f8ea3c640 1 -- start start 2024-09-13T11:53:09.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.410+0000 7f1f8ea3c640 10 monclient: init 2024-09-13T11:53:09.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8ea3c640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:09.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8ea3c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:53:09.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8ea3c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1f880a2980 0x7f1f881364c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8ea3c640 10 monclient(hunting): picked mon.noname-b con 0x7f1f880a2980 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:09.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8ea3c640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f88137a60 0x7f1f88136a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8ea3c640 10 monclient(hunting): picked mon.noname-c con 0x7f1f88137a60 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:09.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8ea3c640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f8813e070 0x7f1f88139c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.440 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f0175639640 1 Processor -- start 2024-09-13T11:53:09.441 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8ea3c640 10 monclient(hunting): picked mon.noname-a con 0x7f1f8813e070 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:09.441 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8ea3c640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.441 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8ea3c640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8806fd80 con 0x7f1f8813e070 2024-09-13T11:53:09.441 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8ea3c640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.441 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8ea3c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1f880a1700 con 0x7f1f880a2980 2024-09-13T11:53:09.441 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8ea3c640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.441 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8ea3c640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f1f8806ff00 con 0x7f1f88137a60 2024-09-13T11:53:09.441 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8ea3c640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:09.441 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8ea3c640 10 monclient(hunting): authenticate will time out at 770.004330s 2024-09-13T11:53:09.442 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f0175639640 1 -- start start 2024-09-13T11:53:09.442 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f0175639640 10 monclient: init 2024-09-13T11:53:09.444 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f877fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f88137a60 0x7f1f88136a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.444 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f877fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f88137a60 0x7f1f88136a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47442/0 (socket says 172.21.15.100:47442) 2024-09-13T11:53:09.444 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f877fe640 1 -- 172.21.15.100:0/3878470693 learned_addr learned my addr 172.21.15.100:0/3878470693 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:09.444 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f877fe640 10 monclient(hunting): get_auth_request con 0x7f1f88137a60 auth_method 0 2024-09-13T11:53:09.444 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.444 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f877fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.444 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.444 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f87fff640 1 --2- 172.21.15.100:0/3878470693 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1f880a2980 0x7f1f881364c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.444 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8cfd9640 1 --2- 172.21.15.100:0/3878470693 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f8813e070 0x7f1f88139c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.444 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f87fff640 10 monclient(hunting): get_auth_request con 0x7f1f880a2980 auth_method 0 2024-09-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f87fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8cfd9640 10 monclient(hunting): get_auth_request con 0x7f1f8813e070 auth_method 0 2024-09-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8cfd9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8cfd9640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8cfd9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8cfd9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8cfd9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8cfd9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.446 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.446 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.446 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f877fe640 10 monclient(hunting): handle_auth_done global_id 4301 payload 824 2024-09-13T11:53:09.446 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f877fe640 1 -- 172.21.15.100:0/3878470693 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1f880a2980 msgr2=0x7f1f881364c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.446 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f877fe640 1 --2- 172.21.15.100:0/3878470693 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1f880a2980 0x7f1f881364c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.446 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f877fe640 1 -- 172.21.15.100:0/3878470693 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f8813e070 msgr2=0x7f1f88139c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.446 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f877fe640 1 --2- 172.21.15.100:0/3878470693 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f8813e070 0x7f1f88139c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.446 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f877fe640 10 monclient: _finish_hunting 0 2024-09-13T11:53:09.446 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f877fe640 1 monclient: found mon.noname-c 2024-09-13T11:53:09.446 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:09.446 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f877fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:53:09.446 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f877fe640 1 -- 172.21.15.100:0/3878470693 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f8813fd40 con 0x7f1f88137a60 2024-09-13T11:53:09.446 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.411+0000 7f1f8cfd9640 1 --2- 172.21.15.100:0/3878470693 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f8813e070 0x7f1f88139c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:09.446 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f1f877fe640 1 --2- 172.21.15.100:0/3878470693 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f88137a60 0x7f1f88136a00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c00c4b0 tx=0x7f1f7c00c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:09.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f0175639640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:09.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f0175639640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:53:09.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f0175639640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f01700a26e0 0x7f0170122790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f0175639640 10 monclient(hunting): picked mon.noname-c con 0x7f01700a26e0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:09.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f0175639640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01700a3040 0x7f01701364d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f0175639640 10 monclient(hunting): picked mon.noname-b con 0x7f01700a3040 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:09.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f0175639640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0170137880 0x7f01701cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f0175639640 10 monclient(hunting): picked mon.noname-a con 0x7f0170137880 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:09.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f0175639640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f0175639640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f01700744f0 con 0x7f0170137880 2024-09-13T11:53:09.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f0175639640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f0175639640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f0170076870 con 0x7f01700a3040 2024-09-13T11:53:09.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f0175639640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f0175639640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f0170074670 con 0x7f01700a26e0 2024-09-13T11:53:09.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f0175639640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:09.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f0175639640 10 monclient(hunting): authenticate will time out at 770.005246s 2024-09-13T11:53:09.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016effd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f01700a26e0 0x7f0170122790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016effd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f01700a26e0 0x7f0170122790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47450/0 (socket says 172.21.15.100:47450) 2024-09-13T11:53:09.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016effd640 1 -- 172.21.15.100:0/570501045 learned_addr learned my addr 172.21.15.100:0/570501045 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:09.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016effd640 10 monclient(hunting): get_auth_request con 0x7f01700a26e0 auth_method 0 2024-09-13T11:53:09.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016f7fe640 1 --2- 172.21.15.100:0/570501045 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0170137880 0x7f01701cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016e7fc640 1 --2- 172.21.15.100:0/570501045 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01700a3040 0x7f01701364d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016f7fe640 10 monclient(hunting): get_auth_request con 0x7f0170137880 auth_method 0 2024-09-13T11:53:09.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016e7fc640 10 monclient(hunting): get_auth_request con 0x7f01700a3040 auth_method 0 2024-09-13T11:53:09.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016effd640 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2024-09-13T11:53:09.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016effd640 1 -- 172.21.15.100:0/570501045 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01700a3040 msgr2=0x7f01701364d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016effd640 1 --2- 172.21.15.100:0/570501045 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01700a3040 0x7f01701364d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016effd640 1 -- 172.21.15.100:0/570501045 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0170137880 msgr2=0x7f01701cc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016effd640 1 --2- 172.21.15.100:0/570501045 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0170137880 0x7f01701cc4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016effd640 10 monclient: _finish_hunting 0 2024-09-13T11:53:09.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016effd640 1 monclient: found mon.noname-c 2024-09-13T11:53:09.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:09.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:53:09.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016effd640 1 -- 172.21.15.100:0/570501045 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0170136ed0 con 0x7f01700a26e0 2024-09-13T11:53:09.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f016f7fe640 1 --2- 172.21.15.100:0/570501045 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0170137880 0x7f01701cc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:09.454 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f1f84ff9640 1 -- 172.21.15.100:0/3878470693 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c018070 con 0x7f1f88137a60 2024-09-13T11:53:09.454 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f1f84ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:09.454 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f1f84ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:09.454 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f1f84ff9640 10 monclient: dump: 2024-09-13T11:53:09.454 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:09.454 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.454 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.455 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.455 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:09.455 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.455 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:09.455 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:09.455 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:09.455 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f1f84ff9640 10 monclient: _finish_auth 0 2024-09-13T11:53:09.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f1f84ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:53:09.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f1f84ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:09.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f1f84ff9640 1 -- 172.21.15.100:0/3878470693 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f7c0074e0 con 0x7f1f88137a60 2024-09-13T11:53:09.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f1f84ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:09.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f1f8ea3c640 5 monclient: authenticate success, global_id 4301 2024-09-13T11:53:09.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f1f8ea3c640 10 monclient: _renew_subs 2024-09-13T11:53:09.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f1f8ea3c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:09.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f1f8ea3c640 1 -- 172.21.15.100:0/3878470693 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f880a8c20 con 0x7f1f88137a60 2024-09-13T11:53:09.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f1f867fc640 4 set_mon_vals no callback set 2024-09-13T11:53:09.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f1f8ea3c640 10 monclient: _renew_subs 2024-09-13T11:53:09.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f1f8ea3c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:09.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f1f8ea3c640 1 -- 172.21.15.100:0/3878470693 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f8813d4a0 con 0x7f1f88137a60 2024-09-13T11:53:09.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f1f84ff9640 1 -- 172.21.15.100:0/3878470693 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c0049e0 con 0x7f1f88137a60 2024-09-13T11:53:09.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f1f84ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:09.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f1f84ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:09.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.412+0000 7f1f84ff9640 10 monclient: dump: 2024-09-13T11:53:09.456 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:09.456 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.456 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.456 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.457 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:09.457 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.457 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:09.457 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:09.457 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:09.457 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.458 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T11:53:09.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f016effd640 1 --2- 172.21.15.100:0/570501045 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f01700a26e0 0x7f0170122790 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0160000c00 tx=0x7f01600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:09.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f01577fe640 1 -- 172.21.15.100:0/570501045 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0160037040 con 0x7f01700a26e0 2024-09-13T11:53:09.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f01577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:09.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f01577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:09.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f01577fe640 10 monclient: dump: 2024-09-13T11:53:09.459 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:09.459 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.459 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.459 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.459 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:09.459 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.459 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:09.459 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:09.459 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:09.459 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f01577fe640 10 monclient: _finish_auth 0 2024-09-13T11:53:09.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f01577fe640 10 monclient: _check_auth_tickets 2024-09-13T11:53:09.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f01577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:09.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f01577fe640 1 -- 172.21.15.100:0/570501045 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0160042070 con 0x7f01700a26e0 2024-09-13T11:53:09.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f01577fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:09.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f01577fe640 1 -- 172.21.15.100:0/570501045 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0160030880 con 0x7f01700a26e0 2024-09-13T11:53:09.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f01577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:09.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f01577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:09.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f01577fe640 10 monclient: dump: 2024-09-13T11:53:09.460 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:09.460 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.460 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.460 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.461 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:09.461 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.461 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:09.461 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:09.461 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:09.461 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.461 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f016d7fa640 4 set_mon_vals no callback set 2024-09-13T11:53:09.461 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f0175639640 5 monclient: authenticate success, global_id 4304 2024-09-13T11:53:09.461 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f0175639640 10 monclient: _renew_subs 2024-09-13T11:53:09.461 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f0175639640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:09.461 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f0175639640 1 -- 172.21.15.100:0/570501045 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0170121c30 con 0x7f01700a26e0 2024-09-13T11:53:09.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f1f8ea3c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:09.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f1f8ea3c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:09.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f1f8ea3c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:09.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f1f8ea3c640 1 -- 172.21.15.100:0/3878470693 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f88137e40 con 0x7f1f88137a60 2024-09-13T11:53:09.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f0175639640 10 monclient: _renew_subs 2024-09-13T11:53:09.463 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f0175639640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:09.463 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f0175639640 1 -- 172.21.15.100:0/570501045 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0170123cb0 con 0x7f01700a26e0 2024-09-13T11:53:09.464 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.413+0000 7f1f84ff9640 1 -- 172.21.15.100:0/3878470693 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f1f7c01c3f0 con 0x7f1f88137a60 2024-09-13T11:53:09.464 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.414+0000 7f1f84ff9640 1 --2- 172.21.15.100:0/3878470693 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f1f7403dfa0 0x7f1f74040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.464 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.414+0000 7f1f84ff9640 1 -- 172.21.15.100:0/3878470693 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7f1f7c051930 con 0x7f1f88137a60 2024-09-13T11:53:09.464 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.414+0000 7f1f87fff640 1 --2- 172.21.15.100:0/3878470693 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f1f7403dfa0 0x7f1f74040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.464 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.414+0000 7f1f87fff640 10 monclient: get_auth_request con 0x7f1f7403dfa0 auth_method 0 2024-09-13T11:53:09.465 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.414+0000 7f01577fe640 1 -- 172.21.15.100:0/570501045 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f016002ad10 con 0x7f01700a26e0 2024-09-13T11:53:09.465 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.414+0000 7f0175639640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:09.465 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.414+0000 7f0175639640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:09.465 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.414+0000 7f0175639640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:09.465 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.414+0000 7f0175639640 1 -- 172.21.15.100:0/570501045 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f017009cfa0 con 0x7f01700a26e0 2024-09-13T11:53:09.465 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.415+0000 7f01577fe640 1 --2- 172.21.15.100:0/570501045 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f015c03dfa0 0x7f015c040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.466 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.415+0000 7f1f87fff640 1 --2- 172.21.15.100:0/3878470693 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f1f7403dfa0 0x7f1f74040460 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1f78000f60 tx=0x7f1f78009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:09.466 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.415+0000 7f01577fe640 1 -- 172.21.15.100:0/570501045 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7f0160072c60 con 0x7f01700a26e0 2024-09-13T11:53:09.466 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.415+0000 7f016e7fc640 1 --2- 172.21.15.100:0/570501045 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f015c03dfa0 0x7f015c040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.467 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.415+0000 7f016e7fc640 10 monclient: get_auth_request con 0x7f015c03dfa0 auth_method 0 2024-09-13T11:53:09.467 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.415+0000 7f016e7fc640 1 --2- 172.21.15.100:0/570501045 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f015c03dfa0 0x7f015c040460 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f01701222a0 tx=0x7f0164016040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:09.467 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T11:53:09.468 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.419+0000 7f1f84ff9640 1 -- 172.21.15.100:0/3878470693 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f1f7c01b360 con 0x7f1f88137a60 2024-09-13T11:53:09.468 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.419+0000 7f1f84ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:09.468 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.419+0000 7f1f84ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:09.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.427+0000 7f01577fe640 1 -- 172.21.15.100:0/570501045 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f0160047460 con 0x7f01700a26e0 2024-09-13T11:53:09.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.427+0000 7f01577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:09.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.427+0000 7f01577fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:09.509 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.507+0000 7f0ffc260640 10 monclient: get_monmap_and_config 2024-09-13T11:53:09.509 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.507+0000 7f0ffc260640 10 monclient: build_initial_monmap 2024-09-13T11:53:09.509 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.507+0000 7f0ffc260640 10 monclient: monmap: 2024-09-13T11:53:09.509 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:09.510 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.510 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:09.508231+0000 2024-09-13T11:53:09.510 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:09.508231+0000 2024-09-13T11:53:09.510 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:09.510 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.510 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:09.510 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:09.510 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:09.510 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:09.510 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:09.510 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.508+0000 7f0ffc260640 1 Processor -- start 2024-09-13T11:53:09.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.508+0000 7f0ffc260640 1 -- start start 2024-09-13T11:53:09.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.508+0000 7f0ffc260640 10 monclient: init 2024-09-13T11:53:09.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.508+0000 7f0ffc260640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:09.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.508+0000 7f0ffc260640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-13T11:53:09.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.508+0000 7f0ffc260640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0ff4137a60 0x7f0ff40a2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ffc260640 10 monclient(hunting): picked mon.noname-c con 0x7f0ff4137a60 addr v2:172.21.15.122:3300/0 2024-09-13T11:53:09.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ffc260640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ff4137e40 0x7f0ff40a4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ffc260640 10 monclient(hunting): picked mon.noname-b con 0x7f0ff4137e40 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:09.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ffc260640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ff4138270 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:09.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ffc260640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ff40a5be0 con 0x7f0ff4138270 2024-09-13T11:53:09.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ffc260640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ffc260640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f0ff40a4410 con 0x7f0ff4137a60 2024-09-13T11:53:09.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ffc260640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ffc260640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f0ff40a76b0 con 0x7f0ff4137e40 2024-09-13T11:53:09.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ffc260640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:09.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ffc260640 10 monclient(hunting): authenticate will time out at 770.102337s 2024-09-13T11:53:09.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff97fb640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ff4137e40 0x7f0ff40a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff97fb640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ff4137e40 0x7f0ff40a4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47462/0 (socket says 172.21.15.100:47462) 2024-09-13T11:53:09.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff9ffc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0ff4137a60 0x7f0ff40a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff97fb640 1 -- 172.21.15.100:0/4183568297 learned_addr learned my addr 172.21.15.100:0/4183568297 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:09.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff97fb640 10 monclient(hunting): get_auth_request con 0x7f0ff4137e40 auth_method 0 2024-09-13T11:53:09.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff97fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff97fb640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff97fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff9ffc640 10 monclient(hunting): get_auth_request con 0x7f0ff4137a60 auth_method 0 2024-09-13T11:53:09.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff9ffc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff9ffc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff9ffc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff97fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff97fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff97fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff9ffc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff9ffc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff9ffc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff9ffc640 10 monclient(hunting): handle_auth_done global_id 4315 payload 274 2024-09-13T11:53:09.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff9ffc640 1 -- 172.21.15.100:0/4183568297 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ff4137e40 msgr2=0x7f0ff40a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:09.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff9ffc640 1 --2- 172.21.15.100:0/4183568297 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ff4137e40 0x7f0ff40a4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff9ffc640 1 -- 172.21.15.100:0/4183568297 >> v1:172.21.15.122:6789/0 conn(0x7f0ff4138270 legacy=0x7f0ff40a4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff9ffc640 10 monclient: _finish_hunting 0 2024-09-13T11:53:09.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff9ffc640 1 monclient: found mon.noname-c 2024-09-13T11:53:09.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff9ffc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:09.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff9ffc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.122:3300/0 2024-09-13T11:53:09.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.509+0000 7f0ff9ffc640 1 -- 172.21.15.100:0/4183568297 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ff40a19e0 con 0x7f0ff4137a60 2024-09-13T11:53:09.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.511+0000 7f0ff9ffc640 1 --2- 172.21.15.100:0/4183568297 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ff4137a60 0x7f0ff40a2980 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0ff40a7880 tx=0x7f0ff002a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff2f52d77b1610af server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:09.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.511+0000 7f0ff8ffa640 1 -- 172.21.15.100:0/4183568297 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0037070 con 0x7f0ff4137a60 2024-09-13T11:53:09.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.511+0000 7f0ff8ffa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:09.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.511+0000 7f0ff8ffa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:09.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.511+0000 7f0ff8ffa640 10 monclient: dump: 2024-09-13T11:53:09.519 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:09.519 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.519 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.519 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.519 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:09.519 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.520 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:09.520 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:09.520 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:09.520 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.520 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.511+0000 7f0ff8ffa640 10 monclient: _finish_auth 0 2024-09-13T11:53:09.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.511+0000 7f0ff8ffa640 10 monclient: _check_auth_tickets 2024-09-13T11:53:09.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.511+0000 7f0ff8ffa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:09.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.511+0000 7f0ff8ffa640 1 -- 172.21.15.100:0/4183568297 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ff002adb0 con 0x7f0ff4137a60 2024-09-13T11:53:09.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.511+0000 7f0ff8ffa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:09.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.511+0000 7f0ff8ffa640 1 -- 172.21.15.100:0/4183568297 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0032660 con 0x7f0ff4137a60 2024-09-13T11:53:09.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.511+0000 7f0ff8ffa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:09.522 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.511+0000 7f0ff8ffa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:53:09.522 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.511+0000 7f0ff8ffa640 10 monclient: dump: 2024-09-13T11:53:09.522 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:09.522 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.522 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.523 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.523 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:09.523 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.523 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:09.523 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:09.523 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:09.524 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.524 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.511+0000 7f0ffc260640 5 monclient: authenticate success, global_id 4315 2024-09-13T11:53:09.524 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.511+0000 7f0ffc260640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:09.524 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.511+0000 7f0ffc260640 4 set_mon_vals no callback set 2024-09-13T11:53:09.534 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.512+0000 7f0ffc260640 10 monclient: shutdown 2024-09-13T11:53:09.535 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.512+0000 7f0ffc260640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:09.535 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.512+0000 7f0ffc260640 1 -- 172.21.15.100:0/4183568297 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ff4137a60 msgr2=0x7f0ff40a2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.535 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.512+0000 7f0ffc260640 1 --2- 172.21.15.100:0/4183568297 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ff4137a60 0x7f0ff40a2980 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0ff40a7880 tx=0x7f0ff002a060 comp rx=0 tx=0).stop 2024-09-13T11:53:09.535 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.512+0000 7f0ffc260640 1 -- 172.21.15.100:0/4183568297 shutdown_connections 2024-09-13T11:53:09.535 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.512+0000 7f0ffc260640 1 --2- 172.21.15.100:0/4183568297 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ff4137e40 0x7f0ff40a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.536 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.512+0000 7f0ffc260640 1 --2- 172.21.15.100:0/4183568297 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ff4137a60 0x7f0ff40a2980 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.536 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.512+0000 7f0ffc260640 1 -- 172.21.15.100:0/4183568297 >> 172.21.15.100:0/4183568297 conn(0x7f0ff40aa580 msgr2=0x7f0ff40a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:09.536 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.512+0000 7f0ffc260640 1 -- 172.21.15.100:0/4183568297 shutdown_connections 2024-09-13T11:53:09.536 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.512+0000 7f0ffc260640 1 -- 172.21.15.100:0/4183568297 wait complete. 2024-09-13T11:53:09.536 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.513+0000 7f0ffc260640 10 monclient: build_initial_monmap 2024-09-13T11:53:09.536 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.513+0000 7f0ffc260640 10 monclient: monmap: 2024-09-13T11:53:09.536 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:09.536 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:09.536 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:09.536 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:09.537 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:09.537 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.537 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:09.537 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:09.537 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:09.537 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.537 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.513+0000 7f0ffc260640 1 Processor -- start 2024-09-13T11:53:09.537 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ffc260640 1 -- start start 2024-09-13T11:53:09.538 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ffc260640 10 monclient: init 2024-09-13T11:53:09.538 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ffc260640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:09.538 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ffc260640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:53:09.538 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ffc260640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ff4137a60 0x7f0ff4133940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.538 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ffc260640 10 monclient(hunting): picked mon.noname-c con 0x7f0ff4137a60 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:09.538 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ffc260640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0ff4137e40 0x7f0ff4133e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.538 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ffc260640 10 monclient(hunting): picked mon.noname-a con 0x7f0ff4137e40 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:09.538 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ffc260640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ff4138270 0x7f0ff4139610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.538 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ffc260640 10 monclient(hunting): picked mon.noname-b con 0x7f0ff4138270 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:09.538 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ffc260640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.538 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ffc260640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff406ff00 con 0x7f0ff4137e40 2024-09-13T11:53:09.538 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ffc260640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.539 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ffc260640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff406fd80 con 0x7f0ff4138270 2024-09-13T11:53:09.539 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ffc260640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.539 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ffc260640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f0ff40a55a0 con 0x7f0ff4137a60 2024-09-13T11:53:09.539 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ffc260640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:09.539 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ffc260640 10 monclient(hunting): authenticate will time out at 770.107677s 2024-09-13T11:53:09.539 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ff9ffc640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ff4137a60 0x7f0ff4133940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.539 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ff97fb640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0ff4137e40 0x7f0ff4133e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.539 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ff97fb640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0ff4137e40 0x7f0ff4133e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49636/0 (socket says 172.21.15.100:49636) 2024-09-13T11:53:09.539 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ff97fb640 1 -- 172.21.15.100:0/3678200315 learned_addr learned my addr 172.21.15.100:0/3678200315 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:09.539 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ff97fb640 10 monclient(hunting): get_auth_request con 0x7f0ff4137e40 auth_method 0 2024-09-13T11:53:09.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ff97fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ff97fb640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ff97fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ff9ffc640 10 monclient(hunting): get_auth_request con 0x7f0ff4137a60 auth_method 0 2024-09-13T11:53:09.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ff9ffc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ff9ffc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.514+0000 7f0ff9ffc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.515+0000 7f0ff97fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.515+0000 7f0ff97fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.515+0000 7f0ff97fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.515+0000 7f0ffa7fd640 1 --2- 172.21.15.100:0/3678200315 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ff4138270 0x7f0ff4139610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.515+0000 7f0ffa7fd640 10 monclient(hunting): get_auth_request con 0x7f0ff4138270 auth_method 0 2024-09-13T11:53:09.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.515+0000 7f0ffa7fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.515+0000 7f0ffa7fd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.515+0000 7f0ffa7fd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.515+0000 7f0ffa7fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.515+0000 7f0ffa7fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.515+0000 7f0ffa7fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.515+0000 7f0ff97fb640 10 monclient(hunting): handle_auth_done global_id 4338 payload 824 2024-09-13T11:53:09.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.515+0000 7f0ff97fb640 1 -- 172.21.15.100:0/3678200315 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ff4137a60 msgr2=0x7f0ff4133940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.515+0000 7f0ff97fb640 1 --2- 172.21.15.100:0/3678200315 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ff4137a60 0x7f0ff4133940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.515+0000 7f0ff97fb640 1 -- 172.21.15.100:0/3678200315 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ff4138270 msgr2=0x7f0ff4139610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.515+0000 7f0ff97fb640 1 --2- 172.21.15.100:0/3678200315 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ff4138270 0x7f0ff4139610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.515+0000 7f0ff97fb640 10 monclient: _finish_hunting 0 2024-09-13T11:53:09.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.515+0000 7f0ff97fb640 1 monclient: found mon.noname-a 2024-09-13T11:53:09.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.515+0000 7f0ff97fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:09.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.515+0000 7f0ff97fb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:09.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.515+0000 7f0ff97fb640 1 -- 172.21.15.100:0/3678200315 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ff413cb00 con 0x7f0ff4137e40 2024-09-13T11:53:09.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.515+0000 7f0ff9ffc640 1 --2- 172.21.15.100:0/3678200315 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ff4137a60 0x7f0ff4133940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:53:09.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.515+0000 7f0ff97fb640 1 --2- 172.21.15.100:0/3678200315 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0ff4137e40 0x7f0ff4133e80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0fe400e8b0 tx=0x7f0fe400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:09.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.516+0000 7f0fea7fc640 1 -- 172.21.15.100:0/3678200315 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe4017070 con 0x7f0ff4137e40 2024-09-13T11:53:09.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.516+0000 7f0fea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:09.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.516+0000 7f0fea7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:09.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.516+0000 7f0fea7fc640 10 monclient: dump: 2024-09-13T11:53:09.543 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:09.543 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.543 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.544 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.544 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:09.544 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.544 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:09.544 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:09.544 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:09.544 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.516+0000 7f0fea7fc640 10 monclient: _finish_auth 0 2024-09-13T11:53:09.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.516+0000 7f0fea7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:53:09.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.516+0000 7f0fea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:09.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.516+0000 7f0fea7fc640 1 -- 172.21.15.100:0/3678200315 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fe400c8e0 con 0x7f0ff4137e40 2024-09-13T11:53:09.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.516+0000 7f0fea7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:09.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.516+0000 7f0fea7fc640 1 -- 172.21.15.100:0/3678200315 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe400ccb0 con 0x7f0ff4137e40 2024-09-13T11:53:09.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.516+0000 7f0fea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:09.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.516+0000 7f0fea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:09.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.516+0000 7f0fea7fc640 10 monclient: dump: 2024-09-13T11:53:09.545 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:09.546 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.546 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.546 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.546 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:09.546 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.546 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:09.546 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:09.546 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:09.546 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.546 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.516+0000 7f0ffc260640 5 monclient: authenticate success, global_id 4338 2024-09-13T11:53:09.547 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.516+0000 7f0ffc260640 10 monclient: _renew_subs 2024-09-13T11:53:09.547 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.516+0000 7f0ffc260640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:09.547 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.516+0000 7f0ffc260640 1 -- 172.21.15.100:0/3678200315 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ff40a1140 con 0x7f0ff4137e40 2024-09-13T11:53:09.547 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.516+0000 7f0ffc260640 10 monclient: _renew_subs 2024-09-13T11:53:09.547 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.516+0000 7f0ffc260640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:09.547 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.516+0000 7f0ffc260640 1 -- 172.21.15.100:0/3678200315 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ff413d3a0 con 0x7f0ff4137e40 2024-09-13T11:53:09.547 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.516+0000 7f0febfff640 4 set_mon_vals no callback set 2024-09-13T11:53:09.547 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.517+0000 7f0ffc260640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:09.547 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.518+0000 7f0ffc260640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:09.547 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.518+0000 7f0ffc260640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:09.548 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.518+0000 7f0ffc260640 1 -- 172.21.15.100:0/3678200315 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ff40a2980 con 0x7f0ff4137e40 2024-09-13T11:53:09.548 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.518+0000 7f0fea7fc640 1 -- 172.21.15.100:0/3678200315 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f0fe401a070 con 0x7f0ff4137e40 2024-09-13T11:53:09.548 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.518+0000 7f0fea7fc640 1 --2- 172.21.15.100:0/3678200315 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f0fd803e020 0x7f0fd80404e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.548 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.518+0000 7f0fea7fc640 1 -- 172.21.15.100:0/3678200315 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7f0fe40524e0 con 0x7f0ff4137e40 2024-09-13T11:53:09.548 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.519+0000 7f0ff9ffc640 1 --2- 172.21.15.100:0/3678200315 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f0fd803e020 0x7f0fd80404e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.548 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.519+0000 7f0ff9ffc640 10 monclient: get_auth_request con 0x7f0fd803e020 auth_method 0 2024-09-13T11:53:09.548 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.519+0000 7f0ff9ffc640 1 --2- 172.21.15.100:0/3678200315 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f0fd803e020 0x7f0fd80404e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0034000 tx=0x7f0ff002e000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:09.549 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.525+0000 7f516f8e4640 10 monclient: get_monmap_and_config 2024-09-13T11:53:09.549 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.525+0000 7f516f8e4640 10 monclient: build_initial_monmap 2024-09-13T11:53:09.549 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.525+0000 7f516f8e4640 10 monclient: monmap: 2024-09-13T11:53:09.550 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:09.550 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.550 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:09.526314+0000 2024-09-13T11:53:09.550 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:09.526314+0000 2024-09-13T11:53:09.550 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:09.550 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.550 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:09.550 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:09.551 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:09.551 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:09.551 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:09.551 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.551 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.526+0000 7f516f8e4640 1 Processor -- start 2024-09-13T11:53:09.551 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.526+0000 7f516f8e4640 1 -- start start 2024-09-13T11:53:09.552 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.526+0000 7f0fea7fc640 1 -- 172.21.15.100:0/3678200315 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f0ff413cb00 con 0x7f0ff4137e40 2024-09-13T11:53:09.552 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.526+0000 7f0fea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:09.552 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.526+0000 7f0fea7fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:09.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.526+0000 7f516f8e4640 10 monclient: init 2024-09-13T11:53:09.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.527+0000 7f516f8e4640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:09.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.527+0000 7f516f8e4640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-13T11:53:09.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.527+0000 7f516f8e4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f516806ea50 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:09.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.527+0000 7f516f8e4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f51680a2860 0x7f516806f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.527+0000 7f516d680640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f516806ea50 0x7f516806ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:58018/0 (socket says 172.21.15.100:58018) 2024-09-13T11:53:09.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.527+0000 7f516d680640 1 -- 172.21.15.100:0/2862568734 learned_addr learned my addr 172.21.15.100:0/2862568734 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:53:09.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.527+0000 7f516f8e4640 10 monclient(hunting): picked mon.noname-c con 0x7f51680a2860 addr v2:172.21.15.122:3300/0 2024-09-13T11:53:09.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.527+0000 7f516f8e4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f516813c9f0 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:09.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.527+0000 7f516f8e4640 1 -- 172.21.15.100:0/2862568734 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f516806fa80 con 0x7f516806ea50 2024-09-13T11:53:09.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.527+0000 7f516f8e4640 1 -- 172.21.15.100:0/2862568734 --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51680a1590 con 0x7f516813c9f0 2024-09-13T11:53:09.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.527+0000 7f516ce7f640 1 --2- 172.21.15.100:0/2862568734 >> v2:172.21.15.122:3300/0 conn(0x7f51680a2860 0x7f516806f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.527+0000 7f516f8e4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.527+0000 7f516f8e4640 1 -- 172.21.15.100:0/2862568734 --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f51680a2cb0 con 0x7f51680a2860 2024-09-13T11:53:09.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.527+0000 7f516f8e4640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:09.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.527+0000 7f516f8e4640 10 monclient(hunting): authenticate will time out at 770.120994s 2024-09-13T11:53:09.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.528+0000 7f516ce7f640 10 monclient(hunting): get_auth_request con 0x7f51680a2860 auth_method 0 2024-09-13T11:53:09.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.528+0000 7f516ce7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.528+0000 7f516ce7f640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.528+0000 7f516ce7f640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.528+0000 7f516ce7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.528+0000 7f516ce7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.528+0000 7f516ce7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.528+0000 7f515ffff640 1 -- 172.21.15.100:0/2862568734 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3221079460 0 0) 0x7f51680a1590 con 0x7f516813c9f0 2024-09-13T11:53:09.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.528+0000 7f516ce7f640 10 monclient(hunting): handle_auth_done global_id 4321 payload 274 2024-09-13T11:53:09.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.528+0000 7f516ce7f640 1 -- 172.21.15.100:0/2862568734 >> v1:172.21.15.122:6789/0 conn(0x7f516813c9f0 legacy=0x7f516813ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.528+0000 7f516ce7f640 1 -- 172.21.15.100:0/2862568734 >> v1:172.21.15.100:6789/0 conn(0x7f516806ea50 legacy=0x7f516806ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.528+0000 7f516ce7f640 10 monclient: _finish_hunting 0 2024-09-13T11:53:09.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.528+0000 7f516ce7f640 1 monclient: found mon.noname-c 2024-09-13T11:53:09.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.528+0000 7f516ce7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:09.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.528+0000 7f516ce7f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.122:3300/0 2024-09-13T11:53:09.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.528+0000 7f516ce7f640 1 -- 172.21.15.100:0/2862568734 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51680a1870 con 0x7f51680a2860 2024-09-13T11:53:09.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.528+0000 7f515ffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:53:09.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.529+0000 7f516ce7f640 1 --2- 172.21.15.100:0/2862568734 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51680a2860 0x7f516806f540 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f51580097a0 tx=0x7f515802a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b25c01ef2398a03a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:09.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.529+0000 7f515ffff640 1 -- 172.21.15.100:0/2862568734 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5158037070 con 0x7f51680a2860 2024-09-13T11:53:09.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.529+0000 7f515ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:09.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.529+0000 7f515ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:09.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.529+0000 7f515ffff640 10 monclient: dump: 2024-09-13T11:53:09.571 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:09.572 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.572 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.572 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.572 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:09.572 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.572 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:09.572 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:09.572 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:09.572 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.529+0000 7f515ffff640 10 monclient: _finish_auth 0 2024-09-13T11:53:09.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.529+0000 7f515ffff640 10 monclient: _check_auth_tickets 2024-09-13T11:53:09.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.529+0000 7f515ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:09.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.529+0000 7f516f8e4640 5 monclient: authenticate success, global_id 4321 2024-09-13T11:53:09.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.529+0000 7f515ffff640 1 -- 172.21.15.100:0/2862568734 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5158032440 con 0x7f51680a2860 2024-09-13T11:53:09.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.529+0000 7f516f8e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:09.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.529+0000 7f515ffff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:09.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.529+0000 7f515ffff640 1 -- 172.21.15.100:0/2862568734 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f515802ae60 con 0x7f51680a2860 2024-09-13T11:53:09.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.529+0000 7f516f8e4640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:09.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.529+0000 7f516f8e4640 4 set_mon_vals no callback set 2024-09-13T11:53:09.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.529+0000 7f515ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:09.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.529+0000 7f515ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:53:09.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.529+0000 7f515ffff640 10 monclient: dump: 2024-09-13T11:53:09.574 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:09.574 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.574 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.574 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.574 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:09.574 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.574 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:09.574 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:09.574 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:09.574 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.530+0000 7f516f8e4640 10 monclient: shutdown 2024-09-13T11:53:09.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.530+0000 7f516f8e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:09.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.530+0000 7f516f8e4640 1 -- 172.21.15.100:0/2862568734 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51680a2860 msgr2=0x7f516806f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.530+0000 7f516f8e4640 1 --2- 172.21.15.100:0/2862568734 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51680a2860 0x7f516806f540 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f51580097a0 tx=0x7f515802a110 comp rx=0 tx=0).stop 2024-09-13T11:53:09.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.530+0000 7f516f8e4640 1 -- 172.21.15.100:0/2862568734 shutdown_connections 2024-09-13T11:53:09.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.530+0000 7f516f8e4640 1 --2- 172.21.15.100:0/2862568734 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51680a2860 0x7f516806f540 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.530+0000 7f516f8e4640 1 -- 172.21.15.100:0/2862568734 >> 172.21.15.100:0/2862568734 conn(0x7f51680a68e0 msgr2=0x7f51680a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:09.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.530+0000 7f516f8e4640 1 -- 172.21.15.100:0/2862568734 shutdown_connections 2024-09-13T11:53:09.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.531+0000 7f516f8e4640 1 -- 172.21.15.100:0/2862568734 wait complete. 2024-09-13T11:53:09.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.531+0000 7f516f8e4640 10 monclient: build_initial_monmap 2024-09-13T11:53:09.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.531+0000 7f516f8e4640 10 monclient: monmap: 2024-09-13T11:53:09.575 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:09.575 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:09.575 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:09.576 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:09.576 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:09.576 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.576 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:09.576 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:09.576 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:09.576 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.532+0000 7f516f8e4640 1 Processor -- start 2024-09-13T11:53:09.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.532+0000 7f516f8e4640 1 -- start start 2024-09-13T11:53:09.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.532+0000 7f516f8e4640 10 monclient: init 2024-09-13T11:53:09.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.533+0000 7f516f8e4640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:09.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.533+0000 7f516f8e4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:53:09.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.533+0000 7f516f8e4640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f516806ea50 0x7f51681cc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.533+0000 7f516d680640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f516806ea50 0x7f51681cc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.533+0000 7f516f8e4640 10 monclient(hunting): picked mon.noname-a con 0x7f516806ea50 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:09.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.533+0000 7f516d680640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f516806ea50 0x7f51681cc8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49640/0 (socket says 172.21.15.100:49640) 2024-09-13T11:53:09.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.533+0000 7f516f8e4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51680a2860 0x7f51681cce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.533+0000 7f516d680640 1 -- 172.21.15.100:0/7398396 learned_addr learned my addr 172.21.15.100:0/7398396 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:09.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.534+0000 7f516f8e4640 10 monclient(hunting): picked mon.noname-b con 0x7f51680a2860 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:09.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.534+0000 7f516f8e4640 1 --2- 172.21.15.100:0/7398396 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f516813c9f0 0x7f516809d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.534+0000 7f516de81640 1 --2- 172.21.15.100:0/7398396 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f516813c9f0 0x7f516809d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.534+0000 7f516ce7f640 1 --2- 172.21.15.100:0/7398396 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51680a2860 0x7f51681cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.534+0000 7f516f8e4640 10 monclient(hunting): picked mon.noname-c con 0x7f516813c9f0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:09.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.534+0000 7f516f8e4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.534+0000 7f516f8e4640 1 -- 172.21.15.100:0/7398396 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f51680a5be0 con 0x7f516806ea50 2024-09-13T11:53:09.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.534+0000 7f516f8e4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.534+0000 7f516f8e4640 1 -- 172.21.15.100:0/7398396 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f51680a5a60 con 0x7f51680a2860 2024-09-13T11:53:09.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.534+0000 7f516f8e4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.534+0000 7f516f8e4640 1 -- 172.21.15.100:0/7398396 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f5168140500 con 0x7f516813c9f0 2024-09-13T11:53:09.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.534+0000 7f516f8e4640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:09.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.534+0000 7f516f8e4640 10 monclient(hunting): authenticate will time out at 770.127665s 2024-09-13T11:53:09.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.534+0000 7f516d680640 10 monclient(hunting): get_auth_request con 0x7f516806ea50 auth_method 0 2024-09-13T11:53:09.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.534+0000 7f516d680640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.534+0000 7f516d680640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.534+0000 7f516d680640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.534+0000 7f516de81640 10 monclient(hunting): get_auth_request con 0x7f516813c9f0 auth_method 0 2024-09-13T11:53:09.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.534+0000 7f516de81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.534+0000 7f516de81640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.534+0000 7f516de81640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.534+0000 7f516de81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.534+0000 7f516de81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.534+0000 7f516de81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.535+0000 7f516d680640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.535+0000 7f516d680640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.535+0000 7f516d680640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.535+0000 7f516de81640 10 monclient(hunting): handle_auth_done global_id 4313 payload 824 2024-09-13T11:53:09.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.535+0000 7f516de81640 1 -- 172.21.15.100:0/7398396 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51680a2860 msgr2=0x7f51681cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.535+0000 7f516de81640 1 --2- 172.21.15.100:0/7398396 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51680a2860 0x7f51681cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.535+0000 7f516de81640 1 -- 172.21.15.100:0/7398396 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f516806ea50 msgr2=0x7f51681cc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.535+0000 7f516de81640 1 --2- 172.21.15.100:0/7398396 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f516806ea50 0x7f51681cc8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.535+0000 7f516de81640 10 monclient: _finish_hunting 0 2024-09-13T11:53:09.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.535+0000 7f516de81640 1 monclient: found mon.noname-c 2024-09-13T11:53:09.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.535+0000 7f516de81640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:09.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.535+0000 7f516de81640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:53:09.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.535+0000 7f516de81640 1 -- 172.21.15.100:0/7398396 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5168120d70 con 0x7f516813c9f0 2024-09-13T11:53:09.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.535+0000 7f516ce7f640 10 monclient: get_auth_request con 0x7f51680a2860 auth_method 0 2024-09-13T11:53:09.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.535+0000 7f516ce7f640 1 --2- 172.21.15.100:0/7398396 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51680a2860 0x7f51681cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:53:09.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.535+0000 7f516d680640 1 --2- 172.21.15.100:0/7398396 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f516806ea50 0x7f51681cc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:09.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.536+0000 7f516de81640 1 --2- 172.21.15.100:0/7398396 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f516813c9f0 0x7f516809d860 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f516000c8b0 tx=0x7f516000cd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:09.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.536+0000 7f515dffb640 1 -- 172.21.15.100:0/7398396 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5160003b70 con 0x7f516813c9f0 2024-09-13T11:53:09.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.536+0000 7f515dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:09.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.536+0000 7f515dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:09.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.536+0000 7f515dffb640 10 monclient: dump: 2024-09-13T11:53:09.581 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:09.581 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.581 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.581 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.581 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:09.581 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.581 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:09.581 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:09.581 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:09.581 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.536+0000 7f515dffb640 10 monclient: _finish_auth 0 2024-09-13T11:53:09.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.536+0000 7f515dffb640 10 monclient: _check_auth_tickets 2024-09-13T11:53:09.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.536+0000 7f515dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:09.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.536+0000 7f515dffb640 1 -- 172.21.15.100:0/7398396 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5160003d10 con 0x7f516813c9f0 2024-09-13T11:53:09.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.536+0000 7f515dffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:09.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.536+0000 7f515dffb640 1 -- 172.21.15.100:0/7398396 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51600052b0 con 0x7f516813c9f0 2024-09-13T11:53:09.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.536+0000 7f515dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:09.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.536+0000 7f515dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:09.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.536+0000 7f515dffb640 10 monclient: dump: 2024-09-13T11:53:09.582 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:09.582 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.582 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.582 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.582 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:09.582 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.582 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:09.583 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:09.583 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:09.583 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.536+0000 7f515f7fe640 4 set_mon_vals no callback set 2024-09-13T11:53:09.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.537+0000 7f516f8e4640 5 monclient: authenticate success, global_id 4313 2024-09-13T11:53:09.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.537+0000 7f516f8e4640 10 monclient: _renew_subs 2024-09-13T11:53:09.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.537+0000 7f516f8e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:09.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.537+0000 7f516f8e4640 1 -- 172.21.15.100:0/7398396 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51680a8c70 con 0x7f516813c9f0 2024-09-13T11:53:09.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.538+0000 7f516f8e4640 10 monclient: _renew_subs 2024-09-13T11:53:09.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.538+0000 7f516f8e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:09.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.538+0000 7f516f8e4640 1 -- 172.21.15.100:0/7398396 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f516809ffe0 con 0x7f516813c9f0 2024-09-13T11:53:09.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.538+0000 7f515dffb640 1 -- 172.21.15.100:0/7398396 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f5160005450 con 0x7f516813c9f0 2024-09-13T11:53:09.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.539+0000 7f515dffb640 1 --2- 172.21.15.100:0/7398396 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f514c03e050 0x7f514c040510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.539+0000 7f515dffb640 1 -- 172.21.15.100:0/7398396 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7f516000adf0 con 0x7f516813c9f0 2024-09-13T11:53:09.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.539+0000 7f516d680640 1 --2- 172.21.15.100:0/7398396 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f514c03e050 0x7f514c040510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.539+0000 7f516d680640 10 monclient: get_auth_request con 0x7f514c03e050 auth_method 0 2024-09-13T11:53:09.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.539+0000 7f516f8e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:09.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.539+0000 7f516f8e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:09.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.540+0000 7f516f8e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:09.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.540+0000 7f516f8e4640 1 -- 172.21.15.100:0/7398396 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f516806ee30 con 0x7f516813c9f0 2024-09-13T11:53:09.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.540+0000 7f516d680640 1 --2- 172.21.15.100:0/7398396 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f514c03e050 0x7f514c040510 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5164000f60 tx=0x7f5164009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:09.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.549+0000 7f515dffb640 1 -- 172.21.15.100:0/7398396 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f5160050810 con 0x7f516813c9f0 2024-09-13T11:53:09.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.549+0000 7f515dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:09.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.549+0000 7f515dffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:09.618 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.616+0000 7fcc20486640 10 monclient: get_monmap_and_config 2024-09-13T11:53:09.618 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.616+0000 7fcc20486640 10 monclient: build_initial_monmap 2024-09-13T11:53:09.618 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.616+0000 7fcc20486640 10 monclient: monmap: 2024-09-13T11:53:09.618 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:09.618 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.618 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:09.617478+0000 2024-09-13T11:53:09.618 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:09.617478+0000 2024-09-13T11:53:09.618 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:09.618 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.618 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:09.618 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:09.618 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:09.619 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:09.619 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:09.619 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.617+0000 7fcc20486640 1 Processor -- start 2024-09-13T11:53:09.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.617+0000 7fcc20486640 1 -- start start 2024-09-13T11:53:09.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.617+0000 7fcc20486640 10 monclient: init 2024-09-13T11:53:09.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.617+0000 7fcc20486640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:09.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.617+0000 7fcc20486640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-13T11:53:09.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc20486640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fcc18137a60 0x7fcc180a2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc20486640 10 monclient(hunting): picked mon.noname-c con 0x7fcc18137a60 addr v2:172.21.15.122:3300/0 2024-09-13T11:53:09.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc20486640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fcc18137e40 0x7fcc180a4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc20486640 10 monclient(hunting): picked mon.noname-a con 0x7fcc18137e40 addr v2:172.21.15.100:3300/0 2024-09-13T11:53:09.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc20486640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc18138250 0x7fcc180a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc20486640 10 monclient(hunting): picked mon.noname-b con 0x7fcc18138250 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:09.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc20486640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc20486640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7fcc180a4410 con 0x7fcc18137e40 2024-09-13T11:53:09.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc20486640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc20486640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7fcc180a5be0 con 0x7fcc18137a60 2024-09-13T11:53:09.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc20486640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc20486640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fcc180a76b0 con 0x7fcc18138250 2024-09-13T11:53:09.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc20486640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:09.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc20486640 10 monclient(hunting): authenticate will time out at 770.211231s 2024-09-13T11:53:09.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1da21640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fcc18137e40 0x7fcc180a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1da21640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fcc18137e40 0x7fcc180a4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49652/0 (socket says 172.21.15.100:49652) 2024-09-13T11:53:09.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1da21640 1 -- 172.21.15.100:0/2156557456 learned_addr learned my addr 172.21.15.100:0/2156557456 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:09.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1da21640 10 monclient(hunting): get_auth_request con 0x7fcc18137e40 auth_method 0 2024-09-13T11:53:09.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1da21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1da21640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1da21640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1ea23640 1 --2- 172.21.15.100:0/2156557456 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc18138250 0x7fcc180a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1ea23640 10 monclient(hunting): get_auth_request con 0x7fcc18138250 auth_method 0 2024-09-13T11:53:09.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1ea23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1ea23640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1ea23640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1ea23640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1ea23640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1ea23640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1e222640 1 --2- 172.21.15.100:0/2156557456 >> v2:172.21.15.122:3300/0 conn(0x7fcc18137a60 0x7fcc180a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1e222640 10 monclient(hunting): get_auth_request con 0x7fcc18137a60 auth_method 0 2024-09-13T11:53:09.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1e222640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1e222640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1e222640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1e222640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1e222640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1e222640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1da21640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1da21640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.634 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1da21640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.634 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1ea23640 10 monclient(hunting): handle_auth_done global_id 4316 payload 274 2024-09-13T11:53:09.634 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1ea23640 1 -- 172.21.15.100:0/2156557456 >> v2:172.21.15.122:3300/0 conn(0x7fcc18137a60 msgr2=0x7fcc180a2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:09.634 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1ea23640 1 --2- 172.21.15.100:0/2156557456 >> v2:172.21.15.122:3300/0 conn(0x7fcc18137a60 0x7fcc180a2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.634 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1ea23640 1 -- 172.21.15.100:0/2156557456 >> v2:172.21.15.100:3300/0 conn(0x7fcc18137e40 msgr2=0x7fcc180a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:09.634 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1ea23640 1 --2- 172.21.15.100:0/2156557456 >> v2:172.21.15.100:3300/0 conn(0x7fcc18137e40 0x7fcc180a4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.634 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1ea23640 10 monclient: _finish_hunting 0 2024-09-13T11:53:09.634 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1ea23640 1 monclient: found mon.noname-b 2024-09-13T11:53:09.634 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1ea23640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:09.634 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1ea23640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:53:09.634 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.618+0000 7fcc1ea23640 1 -- 172.21.15.100:0/2156557456 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc180a19e0 con 0x7fcc18138250 2024-09-13T11:53:09.634 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.619+0000 7fcc1ea23640 1 --2- 172.21.15.100:0/2156557456 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc18138250 0x7fcc180a4e90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcc180ab010 tx=0x7fcc1002a4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5f0b80c85c7050d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:09.634 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.620+0000 7fcc1d220640 1 -- 172.21.15.100:0/2156557456 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10007740 con 0x7fcc18138250 2024-09-13T11:53:09.635 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.620+0000 7fcc1d220640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:09.635 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.620+0000 7fcc1d220640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:09.635 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.620+0000 7fcc1d220640 10 monclient: dump: 2024-09-13T11:53:09.635 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:09.635 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.635 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.635 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.635 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:09.635 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.635 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:09.635 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:09.635 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:09.635 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.635 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.620+0000 7fcc1d220640 10 monclient: _finish_auth 0 2024-09-13T11:53:09.636 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.620+0000 7fcc1d220640 10 monclient: _check_auth_tickets 2024-09-13T11:53:09.636 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.620+0000 7fcc1d220640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:09.636 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.620+0000 7fcc20486640 5 monclient: authenticate success, global_id 4316 2024-09-13T11:53:09.636 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.620+0000 7fcc20486640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:09.636 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.620+0000 7fcc1d220640 1 -- 172.21.15.100:0/2156557456 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc100078e0 con 0x7fcc18138250 2024-09-13T11:53:09.636 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.620+0000 7fcc1d220640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:09.636 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.620+0000 7fcc20486640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:09.636 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.620+0000 7fcc20486640 4 set_mon_vals no callback set 2024-09-13T11:53:09.636 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.620+0000 7fcc1d220640 1 -- 172.21.15.100:0/2156557456 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10007cf0 con 0x7fcc18138250 2024-09-13T11:53:09.636 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.620+0000 7fcc1d220640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:09.636 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.620+0000 7fcc1d220640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:09.636 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.620+0000 7fcc1d220640 10 monclient: dump: 2024-09-13T11:53:09.636 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:09.637 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.637 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.637 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.637 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:09.637 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.637 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:09.637 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:09.637 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:09.637 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.637 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.621+0000 7fcc20486640 10 monclient: shutdown 2024-09-13T11:53:09.637 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.621+0000 7fcc20486640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:09.637 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.621+0000 7fcc20486640 1 -- 172.21.15.100:0/2156557456 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc18138250 msgr2=0x7fcc180a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.637 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.621+0000 7fcc20486640 1 --2- 172.21.15.100:0/2156557456 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc18138250 0x7fcc180a4e90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcc180ab010 tx=0x7fcc1002a4e0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.637 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.621+0000 7fcc20486640 1 -- 172.21.15.100:0/2156557456 shutdown_connections 2024-09-13T11:53:09.638 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.621+0000 7fcc20486640 1 --2- 172.21.15.100:0/2156557456 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc18138250 0x7fcc180a4e90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.638 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.621+0000 7fcc20486640 1 --2- 172.21.15.100:0/2156557456 >> v2:172.21.15.100:3300/0 conn(0x7fcc18137e40 0x7fcc180a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.638 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.621+0000 7fcc20486640 1 --2- 172.21.15.100:0/2156557456 >> v2:172.21.15.122:3300/0 conn(0x7fcc18137a60 0x7fcc180a2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.638 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.621+0000 7fcc20486640 1 -- 172.21.15.100:0/2156557456 >> 172.21.15.100:0/2156557456 conn(0x7fcc180aa580 msgr2=0x7fcc180a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:09.638 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.621+0000 7fcc20486640 1 -- 172.21.15.100:0/2156557456 shutdown_connections 2024-09-13T11:53:09.638 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.621+0000 7fcc20486640 1 -- 172.21.15.100:0/2156557456 wait complete. 2024-09-13T11:53:09.638 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.621+0000 7fcc20486640 10 monclient: build_initial_monmap 2024-09-13T11:53:09.638 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.621+0000 7fcc20486640 10 monclient: monmap: 2024-09-13T11:53:09.638 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:09.638 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:09.638 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:09.638 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:09.638 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:09.638 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.639 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:09.639 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:09.639 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:09.639 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.639 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.622+0000 7fcc20486640 1 Processor -- start 2024-09-13T11:53:09.639 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.622+0000 7fcc20486640 1 -- start start 2024-09-13T11:53:09.639 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.622+0000 7fcc20486640 10 monclient: init 2024-09-13T11:53:09.639 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.622+0000 7fcc20486640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:09.639 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.622+0000 7fcc20486640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:53:09.639 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.622+0000 7fcc20486640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc18137a60 0x7fcc181cc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.639 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.622+0000 7fcc20486640 10 monclient(hunting): picked mon.noname-a con 0x7fcc18137a60 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:09.639 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.622+0000 7fcc20486640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc18137e40 0x7fcc18136750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.639 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.622+0000 7fcc20486640 10 monclient(hunting): picked mon.noname-b con 0x7fcc18137e40 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:09.640 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.622+0000 7fcc20486640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc18138250 0x7fcc18136c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.640 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.622+0000 7fcc20486640 10 monclient(hunting): picked mon.noname-c con 0x7fcc18138250 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:09.640 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.622+0000 7fcc20486640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.640 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.622+0000 7fcc20486640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1806ff00 con 0x7fcc18137a60 2024-09-13T11:53:09.640 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.622+0000 7fcc20486640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.640 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.622+0000 7fcc20486640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1806fd80 con 0x7fcc18137e40 2024-09-13T11:53:09.640 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.622+0000 7fcc20486640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.640 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.622+0000 7fcc20486640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fcc180a53d0 con 0x7fcc18138250 2024-09-13T11:53:09.640 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.622+0000 7fcc20486640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:09.640 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.622+0000 7fcc20486640 10 monclient(hunting): authenticate will time out at 770.215971s 2024-09-13T11:53:09.640 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1e222640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc18137a60 0x7fcc181cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.640 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1e222640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc18137a60 0x7fcc181cc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49666/0 (socket says 172.21.15.100:49666) 2024-09-13T11:53:09.640 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1e222640 1 -- 172.21.15.100:0/1706190112 learned_addr learned my addr 172.21.15.100:0/1706190112 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:09.641 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1e222640 10 monclient(hunting): get_auth_request con 0x7fcc18137a60 auth_method 0 2024-09-13T11:53:09.641 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1e222640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.641 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1e222640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.641 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1e222640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.641 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1ea23640 1 --2- 172.21.15.100:0/1706190112 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc18138250 0x7fcc18136c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.641 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1ea23640 10 monclient(hunting): get_auth_request con 0x7fcc18138250 auth_method 0 2024-09-13T11:53:09.641 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1ea23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.641 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1ea23640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.641 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1ea23640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.641 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1e222640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.641 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1e222640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.641 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1e222640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.641 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1da21640 1 --2- 172.21.15.100:0/1706190112 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc18137e40 0x7fcc18136750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.642 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1da21640 10 monclient(hunting): get_auth_request con 0x7fcc18137e40 auth_method 0 2024-09-13T11:53:09.642 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1da21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.642 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1da21640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.642 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1da21640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.642 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1ea23640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.642 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1ea23640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.642 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1ea23640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.642 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1e222640 10 monclient(hunting): handle_auth_done global_id 4350 payload 824 2024-09-13T11:53:09.642 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1e222640 1 -- 172.21.15.100:0/1706190112 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc18138250 msgr2=0x7fcc18136c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.642 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1e222640 1 --2- 172.21.15.100:0/1706190112 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc18138250 0x7fcc18136c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.642 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1e222640 1 -- 172.21.15.100:0/1706190112 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc18137e40 msgr2=0x7fcc18136750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.642 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1e222640 1 --2- 172.21.15.100:0/1706190112 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc18137e40 0x7fcc18136750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.642 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1e222640 10 monclient: _finish_hunting 0 2024-09-13T11:53:09.642 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1e222640 1 monclient: found mon.noname-a 2024-09-13T11:53:09.643 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1e222640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:09.643 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1e222640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:09.643 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1e222640 1 -- 172.21.15.100:0/1706190112 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc180a1140 con 0x7fcc18137a60 2024-09-13T11:53:09.643 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1da21640 1 --2- 172.21.15.100:0/1706190112 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc18137e40 0x7fcc18136750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:53:09.643 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.623+0000 7fcc1ea23640 1 --2- 172.21.15.100:0/1706190112 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc18138250 0x7fcc18136c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:09.643 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.624+0000 7fcc1e222640 1 --2- 172.21.15.100:0/1706190112 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc18137a60 0x7fcc181cc590 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcc1400e9e0 tx=0x7fcc1400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:09.643 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.624+0000 7fcc0effd640 1 -- 172.21.15.100:0/1706190112 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc14017070 con 0x7fcc18137a60 2024-09-13T11:53:09.643 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.624+0000 7fcc0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:09.643 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.624+0000 7fcc0effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:09.643 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.624+0000 7fcc0effd640 10 monclient: dump: 2024-09-13T11:53:09.643 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:09.643 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.643 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.644 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.644 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:09.644 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.644 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:09.644 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:09.644 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:09.644 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.644 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.624+0000 7fcc0effd640 10 monclient: _finish_auth 0 2024-09-13T11:53:09.644 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.624+0000 7fcc0effd640 10 monclient: _check_auth_tickets 2024-09-13T11:53:09.644 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.624+0000 7fcc0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:09.644 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.624+0000 7fcc0effd640 1 -- 172.21.15.100:0/1706190112 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc1400c9d0 con 0x7fcc18137a60 2024-09-13T11:53:09.644 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.624+0000 7fcc0effd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:09.644 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.624+0000 7fcc1ca1f640 4 set_mon_vals no callback set 2024-09-13T11:53:09.645 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.624+0000 7fcc0effd640 1 -- 172.21.15.100:0/1706190112 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1400cdc0 con 0x7fcc18137a60 2024-09-13T11:53:09.645 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.624+0000 7fcc0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:09.645 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.624+0000 7fcc0effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:09.645 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.624+0000 7fcc0effd640 10 monclient: dump: 2024-09-13T11:53:09.645 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:09.645 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.645 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.645 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.645 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:09.645 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.645 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:09.645 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:09.645 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:09.645 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.646 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.624+0000 7fcc20486640 5 monclient: authenticate success, global_id 4350 2024-09-13T11:53:09.646 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.624+0000 7fcc20486640 10 monclient: _renew_subs 2024-09-13T11:53:09.646 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.624+0000 7fcc20486640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:09.646 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.625+0000 7fcc20486640 1 -- 172.21.15.100:0/1706190112 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc18128150 con 0x7fcc18137a60 2024-09-13T11:53:09.646 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.625+0000 7fcc20486640 10 monclient: _renew_subs 2024-09-13T11:53:09.646 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.625+0000 7fcc20486640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:09.646 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.625+0000 7fcc20486640 1 -- 172.21.15.100:0/1706190112 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc18125df0 con 0x7fcc18137a60 2024-09-13T11:53:09.646 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.626+0000 7fcc20486640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:09.646 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.626+0000 7fcc20486640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:09.646 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.626+0000 7fcc20486640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:09.646 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.626+0000 7fcc20486640 1 -- 172.21.15.100:0/1706190112 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc180a2980 con 0x7fcc18137a60 2024-09-13T11:53:09.646 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.626+0000 7fcc0effd640 1 -- 172.21.15.100:0/1706190112 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fcc1401a070 con 0x7fcc18137a60 2024-09-13T11:53:09.647 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.626+0000 7fcc0effd640 1 --2- 172.21.15.100:0/1706190112 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fcbfc03dfd0 0x7fcbfc040490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.647 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.626+0000 7fcc0effd640 1 -- 172.21.15.100:0/1706190112 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7fcc14052610 con 0x7fcc18137a60 2024-09-13T11:53:09.647 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.627+0000 7fcc1da21640 1 --2- 172.21.15.100:0/1706190112 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fcbfc03dfd0 0x7fcbfc040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.647 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.627+0000 7fcc1da21640 10 monclient: get_auth_request con 0x7fcbfc03dfd0 auth_method 0 2024-09-13T11:53:09.647 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.627+0000 7fcc1da21640 1 --2- 172.21.15.100:0/1706190112 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fcbfc03dfd0 0x7fcbfc040490 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcc18128640 tx=0x7fcc08008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:09.648 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.631+0000 7fcc0effd640 1 -- 172.21.15.100:0/1706190112 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fcc180a1140 con 0x7fcc18137a60 2024-09-13T11:53:09.648 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.631+0000 7fcc0effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:09.648 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.631+0000 7fcc0effd640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:09.648 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.633+0000 7f0175639640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-13T11:53:09.648 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.633+0000 7f0175639640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-13T11:53:09.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.633+0000 7f0175639640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:09.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.633+0000 7f0175639640 1 -- 172.21.15.100:0/570501045 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f0170069100 con 0x7f01700a26e0 2024-09-13T11:53:09.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.634+0000 7f01577fe640 1 -- 172.21.15.100:0/570501045 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f016003a390 con 0x7f01700a26e0 2024-09-13T11:53:09.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.634+0000 7f01577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-13T11:53:09.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.634+0000 7f01577fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:09.649 INFO:teuthology.orchestra.run.smithi100.stdout:51539607557 2024-09-13T11:53:09.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.636+0000 7f0175639640 1 -- 172.21.15.100:0/570501045 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f015c03dfa0 msgr2=0x7f015c040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.636+0000 7f0175639640 1 --2- 172.21.15.100:0/570501045 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f015c03dfa0 0x7f015c040460 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f01701222a0 tx=0x7f0164016040 comp rx=0 tx=0).stop 2024-09-13T11:53:09.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.636+0000 7f0175639640 10 monclient: shutdown 2024-09-13T11:53:09.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.636+0000 7f0175639640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:09.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.636+0000 7f0175639640 1 -- 172.21.15.100:0/570501045 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f01700a26e0 msgr2=0x7f0170122790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.636+0000 7f0175639640 1 --2- 172.21.15.100:0/570501045 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f01700a26e0 0x7f0170122790 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0160000c00 tx=0x7f01600046c0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.636+0000 7f0175639640 1 -- 172.21.15.100:0/570501045 shutdown_connections 2024-09-13T11:53:09.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.636+0000 7f0175639640 1 --2- 172.21.15.100:0/570501045 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f015c03dfa0 0x7f015c040460 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.636+0000 7f0175639640 1 --2- 172.21.15.100:0/570501045 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0170137880 0x7f01701cc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.636+0000 7f0175639640 1 --2- 172.21.15.100:0/570501045 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01700a3040 0x7f01701364d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.636+0000 7f0175639640 1 --2- 172.21.15.100:0/570501045 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f01700a26e0 0x7f0170122790 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.636+0000 7f0175639640 1 -- 172.21.15.100:0/570501045 >> 172.21.15.100:0/570501045 conn(0x7f01700a37d0 msgr2=0x7f01701bebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:09.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.637+0000 7f0175639640 1 -- 172.21.15.100:0/570501045 shutdown_connections 2024-09-13T11:53:09.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.637+0000 7f0175639640 1 -- 172.21.15.100:0/570501045 wait complete. 2024-09-13T11:53:09.651 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.3 2024-09-13T11:53:09.652 DEBUG:teuthology.parallel:result is None 2024-09-13T11:53:09.661 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.659+0000 7fdd25ff3640 10 monclient: get_monmap_and_config 2024-09-13T11:53:09.661 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.659+0000 7fdd25ff3640 10 monclient: build_initial_monmap 2024-09-13T11:53:09.661 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.659+0000 7fdd25ff3640 10 monclient: monmap: 2024-09-13T11:53:09.661 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:09.661 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.661 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:09.660471+0000 2024-09-13T11:53:09.661 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:09.660471+0000 2024-09-13T11:53:09.661 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:09.661 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.661 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:09.661 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:09.661 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:09.662 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:09.662 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:09.662 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.660+0000 7fdd25ff3640 1 Processor -- start 2024-09-13T11:53:09.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.660+0000 7fdd25ff3640 1 -- start start 2024-09-13T11:53:09.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.660+0000 7fdd25ff3640 10 monclient: init 2024-09-13T11:53:09.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd25ff3640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:09.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd25ff3640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-13T11:53:09.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd25ff3640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fdd2006ea50 0x7fdd200a2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd25ff3640 10 monclient(hunting): picked mon.noname-a con 0x7fdd2006ea50 addr v2:172.21.15.100:3300/0 2024-09-13T11:53:09.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd25ff3640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd2006ee30 0x7fdd2006f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd25ff3640 10 monclient(hunting): picked mon.noname-b con 0x7fdd2006ee30 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:09.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd25ff3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdd2006f790 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:09.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd25ff3640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd200a1590 con 0x7fdd2006f790 2024-09-13T11:53:09.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd25ff3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd25ff3640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7fdd200a1870 con 0x7fdd2006ea50 2024-09-13T11:53:09.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd25ff3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd25ff3640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fdd200a1b50 con 0x7fdd2006ee30 2024-09-13T11:53:09.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd25ff3640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:09.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd1ffff640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fdd2006f790 0x7fdd201375a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:58026/0 (socket says 172.21.15.100:58026) 2024-09-13T11:53:09.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd1ffff640 1 -- 172.21.15.100:0/3116078025 learned_addr learned my addr 172.21.15.100:0/3116078025 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:53:09.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd1f7fe640 1 --2- 172.21.15.100:0/3116078025 >> v2:172.21.15.100:3300/0 conn(0x7fdd2006ea50 0x7fdd200a2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd1effd640 1 --2- 172.21.15.100:0/3116078025 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd2006ee30 0x7fdd2006f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd25ff3640 10 monclient(hunting): authenticate will time out at 770.254779s 2024-09-13T11:53:09.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd1effd640 10 monclient(hunting): get_auth_request con 0x7fdd2006ee30 auth_method 0 2024-09-13T11:53:09.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd1effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd1f7fe640 10 monclient(hunting): get_auth_request con 0x7fdd2006ea50 auth_method 0 2024-09-13T11:53:09.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd1e7fc640 1 -- 172.21.15.100:0/3116078025 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4132550472 0 0) 0x7fdd200a1590 con 0x7fdd2006f790 2024-09-13T11:53:09.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd1e7fc640 10 monclient(hunting): _negotiate 2024-09-13T11:53:09.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd1e7fc640 10 monclient(hunting): my global_id is 4353 2024-09-13T11:53:09.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd1e7fc640 1 -- 172.21.15.100:0/3116078025 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd08003520 con 0x7fdd2006f790 2024-09-13T11:53:09.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.661+0000 7fdd1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.662+0000 7fdd1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.662+0000 7fdd1effd640 10 monclient(hunting): handle_auth_done global_id 4322 payload 274 2024-09-13T11:53:09.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.662+0000 7fdd1effd640 1 -- 172.21.15.100:0/3116078025 >> v2:172.21.15.100:3300/0 conn(0x7fdd2006ea50 msgr2=0x7fdd200a2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:09.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.662+0000 7fdd1effd640 1 --2- 172.21.15.100:0/3116078025 >> v2:172.21.15.100:3300/0 conn(0x7fdd2006ea50 0x7fdd200a2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.662+0000 7fdd1effd640 1 -- 172.21.15.100:0/3116078025 >> v1:172.21.15.100:6789/0 conn(0x7fdd2006f790 legacy=0x7fdd201375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.662+0000 7fdd1effd640 10 monclient: _finish_hunting 0 2024-09-13T11:53:09.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.662+0000 7fdd1effd640 1 monclient: found mon.noname-b 2024-09-13T11:53:09.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.662+0000 7fdd1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:09.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.662+0000 7fdd1effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:53:09.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.662+0000 7fdd1effd640 1 -- 172.21.15.100:0/3116078025 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd2006fca0 con 0x7fdd2006ee30 2024-09-13T11:53:09.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.663+0000 7fdd1effd640 1 --2- 172.21.15.100:0/3116078025 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd2006ee30 0x7fdd2006f250 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdd140097a0 tx=0x7fdd14029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85c9ad1cdfe3f483 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:09.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.663+0000 7fdd1e7fc640 1 -- 172.21.15.100:0/3116078025 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd14037070 con 0x7fdd2006ee30 2024-09-13T11:53:09.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.663+0000 7fdd1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:09.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.663+0000 7fdd1e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:09.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.663+0000 7fdd1e7fc640 10 monclient: dump: 2024-09-13T11:53:09.669 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:09.669 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.669 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.669 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.669 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:09.669 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.669 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:09.669 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:09.669 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:09.670 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.663+0000 7fdd1e7fc640 10 monclient: _finish_auth 0 2024-09-13T11:53:09.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.663+0000 7fdd1e7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:53:09.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.663+0000 7fdd1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:09.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.663+0000 7fdd1e7fc640 1 -- 172.21.15.100:0/3116078025 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd1402aa60 con 0x7fdd2006ee30 2024-09-13T11:53:09.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.663+0000 7fdd25ff3640 5 monclient: authenticate success, global_id 4322 2024-09-13T11:53:09.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.663+0000 7fdd25ff3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:09.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.663+0000 7fdd1e7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:09.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.663+0000 7fdd1e7fc640 1 -- 172.21.15.100:0/3116078025 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd18001a70 con 0x7fdd2006ee30 2024-09-13T11:53:09.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.663+0000 7fdd1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:09.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.663+0000 7fdd1e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:09.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.663+0000 7fdd1e7fc640 10 monclient: dump: 2024-09-13T11:53:09.670 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:09.671 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.671 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.671 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.671 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:09.671 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.671 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:09.671 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:09.671 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:09.671 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.671 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.663+0000 7fdd25ff3640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:09.671 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.663+0000 7fdd25ff3640 4 set_mon_vals no callback set 2024-09-13T11:53:09.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.664+0000 7fdd25ff3640 10 monclient: shutdown 2024-09-13T11:53:09.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.664+0000 7fdd25ff3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:09.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.664+0000 7fdd25ff3640 1 -- 172.21.15.100:0/3116078025 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd2006ee30 msgr2=0x7fdd2006f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.664+0000 7fdd25ff3640 1 --2- 172.21.15.100:0/3116078025 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd2006ee30 0x7fdd2006f250 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdd140097a0 tx=0x7fdd14029fd0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.664+0000 7fdd25ff3640 1 -- 172.21.15.100:0/3116078025 shutdown_connections 2024-09-13T11:53:09.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.664+0000 7fdd25ff3640 1 --2- 172.21.15.100:0/3116078025 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd2006ee30 0x7fdd2006f250 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.664+0000 7fdd25ff3640 1 --2- 172.21.15.100:0/3116078025 >> v2:172.21.15.100:3300/0 conn(0x7fdd2006ea50 0x7fdd200a2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.664+0000 7fdd25ff3640 1 -- 172.21.15.100:0/3116078025 >> 172.21.15.100:0/3116078025 conn(0x7fdd200a68e0 msgr2=0x7fdd200a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:09.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.664+0000 7fdd25ff3640 1 -- 172.21.15.100:0/3116078025 shutdown_connections 2024-09-13T11:53:09.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.664+0000 7fdd25ff3640 1 -- 172.21.15.100:0/3116078025 wait complete. 2024-09-13T11:53:09.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.665+0000 7fdd25ff3640 10 monclient: build_initial_monmap 2024-09-13T11:53:09.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.665+0000 7fdd25ff3640 10 monclient: monmap: 2024-09-13T11:53:09.679 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:09.679 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:09.680 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:09.680 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:09.680 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:09.680 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.680 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:09.680 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:09.680 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:09.680 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.665+0000 7fdd25ff3640 1 Processor -- start 2024-09-13T11:53:09.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.666+0000 7fdd25ff3640 1 -- start start 2024-09-13T11:53:09.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.666+0000 7fdd25ff3640 10 monclient: init 2024-09-13T11:53:09.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.666+0000 7fdd25ff3640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:09.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.666+0000 7fdd25ff3640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:53:09.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.666+0000 7fdd25ff3640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd2006ea50 0x7fdd201261f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.666+0000 7fdd25ff3640 10 monclient(hunting): picked mon.noname-a con 0x7fdd2006ea50 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:09.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.666+0000 7fdd25ff3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd2006ee30 0x7fdd20123d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.666+0000 7fdd25ff3640 10 monclient(hunting): picked mon.noname-b con 0x7fdd2006ee30 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:09.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.666+0000 7fdd25ff3640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd2006f790 0x7fdd20124260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.666+0000 7fdd25ff3640 10 monclient(hunting): picked mon.noname-c con 0x7fdd2006f790 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:09.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.666+0000 7fdd25ff3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.666+0000 7fdd25ff3640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fdd20137e30 con 0x7fdd2006ea50 2024-09-13T11:53:09.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd25ff3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd25ff3640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fdd20137cb0 con 0x7fdd2006ee30 2024-09-13T11:53:09.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd25ff3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:09.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1ffff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd2006f790 0x7fdd20124260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd25ff3640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fdd20137fb0 con 0x7fdd2006f790 2024-09-13T11:53:09.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd25ff3640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:09.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd25ff3640 10 monclient(hunting): authenticate will time out at 770.260092s 2024-09-13T11:53:09.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1ffff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd2006f790 0x7fdd20124260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47504/0 (socket says 172.21.15.100:47504) 2024-09-13T11:53:09.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1ffff640 1 -- 172.21.15.100:0/170894101 learned_addr learned my addr 172.21.15.100:0/170894101 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:09.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1f7fe640 1 --2- 172.21.15.100:0/170894101 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd2006ea50 0x7fdd201261f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1effd640 1 --2- 172.21.15.100:0/170894101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd2006ee30 0x7fdd20123d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1f7fe640 10 monclient(hunting): get_auth_request con 0x7fdd2006ea50 auth_method 0 2024-09-13T11:53:09.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1ffff640 10 monclient(hunting): get_auth_request con 0x7fdd2006f790 auth_method 0 2024-09-13T11:53:09.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1effd640 10 monclient(hunting): get_auth_request con 0x7fdd2006ee30 auth_method 0 2024-09-13T11:53:09.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:09.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:09.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:09.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:09.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:09.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:09.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1effd640 10 monclient(hunting): handle_auth_done global_id 4333 payload 824 2024-09-13T11:53:09.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1effd640 1 -- 172.21.15.100:0/170894101 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd2006f790 msgr2=0x7fdd20124260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1effd640 1 --2- 172.21.15.100:0/170894101 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd2006f790 0x7fdd20124260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1effd640 1 -- 172.21.15.100:0/170894101 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd2006ea50 msgr2=0x7fdd201261f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1effd640 1 --2- 172.21.15.100:0/170894101 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd2006ea50 0x7fdd201261f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1effd640 10 monclient: _finish_hunting 0 2024-09-13T11:53:09.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1effd640 1 monclient: found mon.noname-b 2024-09-13T11:53:09.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:09.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.122:3300/0 2024-09-13T11:53:09.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1effd640 1 -- 172.21.15.100:0/170894101 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd20127710 con 0x7fdd2006ee30 2024-09-13T11:53:09.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1effd640 1 --2- 172.21.15.100:0/170894101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd2006ee30 0x7fdd20123d20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdd14031000 tx=0x7fdd140046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:09.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd07fff640 1 -- 172.21.15.100:0/170894101 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd14037040 con 0x7fdd2006ee30 2024-09-13T11:53:09.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:09.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd07fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:09.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd07fff640 10 monclient: dump: 2024-09-13T11:53:09.685 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:09.685 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.685 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.685 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.685 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:09.685 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.685 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:09.685 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:09.686 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:09.686 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd07fff640 10 monclient: _finish_auth 0 2024-09-13T11:53:09.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd07fff640 10 monclient: _check_auth_tickets 2024-09-13T11:53:09.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:09.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd07fff640 1 -- 172.21.15.100:0/170894101 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd14032d30 con 0x7fdd2006ee30 2024-09-13T11:53:09.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd07fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:09.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd07fff640 1 -- 172.21.15.100:0/170894101 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd14004230 con 0x7fdd2006ee30 2024-09-13T11:53:09.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:09.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd07fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:53:09.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd1dffb640 4 set_mon_vals no callback set 2024-09-13T11:53:09.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.667+0000 7fdd07fff640 10 monclient: dump: 2024-09-13T11:53:09.686 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:09.687 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:09.687 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.687 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:09.687 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:09.687 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:09.687 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:09.687 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:09.687 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:09.687 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:09.687 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.668+0000 7fdd25ff3640 5 monclient: authenticate success, global_id 4333 2024-09-13T11:53:09.687 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.668+0000 7fdd25ff3640 10 monclient: _renew_subs 2024-09-13T11:53:09.687 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.668+0000 7fdd25ff3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:53:09.687 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.668+0000 7fdd25ff3640 1 -- 172.21.15.100:0/170894101 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd201279f0 con 0x7fdd2006ee30 2024-09-13T11:53:09.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.668+0000 7fdd25ff3640 10 monclient: _renew_subs 2024-09-13T11:53:09.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.668+0000 7fdd25ff3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:53:09.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.668+0000 7fdd25ff3640 1 -- 172.21.15.100:0/170894101 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd20126b80 con 0x7fdd2006ee30 2024-09-13T11:53:09.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.669+0000 7fdd25ff3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:09.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.669+0000 7fdd25ff3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:09.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.669+0000 7fdd25ff3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:53:09.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.669+0000 7fdd25ff3640 1 -- 172.21.15.100:0/170894101 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd200a2860 con 0x7fdd2006ee30 2024-09-13T11:53:09.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.669+0000 7fdd07fff640 1 -- 172.21.15.100:0/170894101 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fdd140043d0 con 0x7fdd2006ee30 2024-09-13T11:53:09.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.669+0000 7fdd07fff640 1 --2- 172.21.15.100:0/170894101 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fdd0c03dff0 0x7fdd0c0404b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:09.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.669+0000 7fdd07fff640 1 -- 172.21.15.100:0/170894101 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7fdd14073150 con 0x7fdd2006ee30 2024-09-13T11:53:09.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.669+0000 7fdd1f7fe640 1 --2- 172.21.15.100:0/170894101 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fdd0c03dff0 0x7fdd0c0404b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:09.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.669+0000 7fdd1f7fe640 10 monclient: get_auth_request con 0x7fdd0c03dff0 auth_method 0 2024-09-13T11:53:09.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.670+0000 7fdd1f7fe640 1 --2- 172.21.15.100:0/170894101 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fdd0c03dff0 0x7fdd0c0404b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdd10009b40 tx=0x7fdd10009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:09.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.674+0000 7fdd07fff640 1 -- 172.21.15.100:0/170894101 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fdd14071c00 con 0x7fdd2006ee30 2024-09-13T11:53:09.689 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.674+0000 7fdd07fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:09.689 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.674+0000 7fdd07fff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:09.689 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.678+0000 7f1f8ea3c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-13T11:53:09.689 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.678+0000 7f1f8ea3c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-13T11:53:09.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.678+0000 7f1f8ea3c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:09.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.678+0000 7f1f8ea3c640 1 -- 172.21.15.100:0/3878470693 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f1f8813bde0 con 0x7f1f88137a60 2024-09-13T11:53:09.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.678+0000 7f1f84ff9640 1 -- 172.21.15.100:0/3878470693 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1f7c010300 con 0x7f1f88137a60 2024-09-13T11:53:09.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.678+0000 7f1f84ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-13T11:53:09.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.678+0000 7f1f84ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:09.690 INFO:teuthology.orchestra.run.smithi100.stdout:47244640261 2024-09-13T11:53:09.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.680+0000 7f1f8ea3c640 1 -- 172.21.15.100:0/3878470693 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f1f7403dfa0 msgr2=0x7f1f74040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.680+0000 7f1f8ea3c640 1 --2- 172.21.15.100:0/3878470693 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f1f7403dfa0 0x7f1f74040460 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1f78000f60 tx=0x7f1f78009040 comp rx=0 tx=0).stop 2024-09-13T11:53:09.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.680+0000 7f1f8ea3c640 10 monclient: shutdown 2024-09-13T11:53:09.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.680+0000 7f1f8ea3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:09.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.680+0000 7f1f8ea3c640 1 -- 172.21.15.100:0/3878470693 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f88137a60 msgr2=0x7f1f88136a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.680+0000 7f1f8ea3c640 1 --2- 172.21.15.100:0/3878470693 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f88137a60 0x7f1f88136a00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c00c4b0 tx=0x7f1f7c00c970 comp rx=0 tx=0).stop 2024-09-13T11:53:09.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.681+0000 7f1f8ea3c640 1 -- 172.21.15.100:0/3878470693 shutdown_connections 2024-09-13T11:53:09.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.681+0000 7f1f8ea3c640 1 --2- 172.21.15.100:0/3878470693 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f1f7403dfa0 0x7f1f74040460 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.681+0000 7f1f8ea3c640 1 --2- 172.21.15.100:0/3878470693 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f8813e070 0x7f1f88139c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.681+0000 7f1f8ea3c640 1 --2- 172.21.15.100:0/3878470693 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f88137a60 0x7f1f88136a00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.681+0000 7f1f8ea3c640 1 --2- 172.21.15.100:0/3878470693 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1f880a2980 0x7f1f881364c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.681+0000 7f1f8ea3c640 1 -- 172.21.15.100:0/3878470693 >> 172.21.15.100:0/3878470693 conn(0x7f1f880aa580 msgr2=0x7f1f8813e450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:09.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.682+0000 7f1f8ea3c640 1 -- 172.21.15.100:0/3878470693 shutdown_connections 2024-09-13T11:53:09.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.682+0000 7f1f8ea3c640 1 -- 172.21.15.100:0/3878470693 wait complete. 2024-09-13T11:53:09.694 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.2 2024-09-13T11:53:09.694 DEBUG:teuthology.parallel:result is None 2024-09-13T11:53:09.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.809+0000 7f0ffc260640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-13T11:53:09.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.809+0000 7f0ffc260640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-13T11:53:09.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.809+0000 7f0ffc260640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:09.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.809+0000 7f0ffc260640 1 -- 172.21.15.100:0/3678200315 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f0ff406a0f0 con 0x7f0ff4137e40 2024-09-13T11:53:09.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.809+0000 7f0fea7fc640 1 -- 172.21.15.100:0/3678200315 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0fe4011350 con 0x7f0ff4137e40 2024-09-13T11:53:09.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.809+0000 7f0fea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-13T11:53:09.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.809+0000 7f0fea7fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:09.811 INFO:teuthology.orchestra.run.smithi100.stdout:47244640262 2024-09-13T11:53:09.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.811+0000 7f0ffc260640 1 -- 172.21.15.100:0/3678200315 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f0fd803e020 msgr2=0x7f0fd80404e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.811+0000 7f0ffc260640 1 --2- 172.21.15.100:0/3678200315 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f0fd803e020 0x7f0fd80404e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0034000 tx=0x7f0ff002e000 comp rx=0 tx=0).stop 2024-09-13T11:53:09.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.811+0000 7f0ffc260640 10 monclient: shutdown 2024-09-13T11:53:09.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.811+0000 7f0ffc260640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:09.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.811+0000 7f0ffc260640 1 -- 172.21.15.100:0/3678200315 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0ff4137e40 msgr2=0x7f0ff4133e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.811+0000 7f0ffc260640 1 --2- 172.21.15.100:0/3678200315 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0ff4137e40 0x7f0ff4133e80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0fe400e8b0 tx=0x7f0fe400ed70 comp rx=0 tx=0).stop 2024-09-13T11:53:09.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.811+0000 7f0ffc260640 1 -- 172.21.15.100:0/3678200315 shutdown_connections 2024-09-13T11:53:09.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.811+0000 7f0ffc260640 1 --2- 172.21.15.100:0/3678200315 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f0fd803e020 0x7f0fd80404e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.811+0000 7f0ffc260640 1 --2- 172.21.15.100:0/3678200315 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0ff4137e40 0x7f0ff4133e80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.811+0000 7f0ffc260640 1 --2- 172.21.15.100:0/3678200315 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ff4138270 0x7f0ff4139610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.811+0000 7f0ffc260640 1 --2- 172.21.15.100:0/3678200315 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ff4137a60 0x7f0ff4133940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.811+0000 7f0ffc260640 1 -- 172.21.15.100:0/3678200315 >> 172.21.15.100:0/3678200315 conn(0x7f0ff40aa580 msgr2=0x7f0ff40a8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:09.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.812+0000 7f0ffc260640 1 -- 172.21.15.100:0/3678200315 shutdown_connections 2024-09-13T11:53:09.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.812+0000 7f0ffc260640 1 -- 172.21.15.100:0/3678200315 wait complete. 2024-09-13T11:53:09.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.825+0000 7fcc20486640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-13T11:53:09.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.825+0000 7fcc20486640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-13T11:53:09.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.825+0000 7fcc20486640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:09.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.825+0000 7fcc20486640 1 -- 172.21.15.100:0/1706190112 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fcc18127dd0 con 0x7fcc18137a60 2024-09-13T11:53:09.827 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640262 for osd.5 2024-09-13T11:53:09.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.825+0000 7fcc0effd640 1 -- 172.21.15.100:0/1706190112 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcc14013c70 con 0x7fcc18137a60 2024-09-13T11:53:09.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.825+0000 7fcc0effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-13T11:53:09.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.825+0000 7fcc0effd640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:09.828 DEBUG:teuthology.parallel:result is None 2024-09-13T11:53:09.828 INFO:teuthology.orchestra.run.smithi100.stdout:47244640262 2024-09-13T11:53:09.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.827+0000 7f516f8e4640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-13T11:53:09.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.827+0000 7f516f8e4640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-13T11:53:09.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.827+0000 7f516f8e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:09.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.827+0000 7f516f8e4640 1 -- 172.21.15.100:0/7398396 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f5168069100 con 0x7f516813c9f0 2024-09-13T11:53:09.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.827+0000 7f515dffb640 1 -- 172.21.15.100:0/7398396 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5160005740 con 0x7f516813c9f0 2024-09-13T11:53:09.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.827+0000 7f515dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-13T11:53:09.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.827+0000 7f515dffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:09.830 INFO:teuthology.orchestra.run.smithi100.stdout:51539607558 2024-09-13T11:53:09.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.828+0000 7fcc20486640 1 -- 172.21.15.100:0/1706190112 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fcbfc03dfd0 msgr2=0x7fcbfc040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.828+0000 7fcc20486640 1 --2- 172.21.15.100:0/1706190112 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fcbfc03dfd0 0x7fcbfc040490 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcc18128640 tx=0x7fcc08008040 comp rx=0 tx=0).stop 2024-09-13T11:53:09.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.828+0000 7fcc20486640 10 monclient: shutdown 2024-09-13T11:53:09.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.828+0000 7fcc20486640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:09.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.828+0000 7fcc20486640 1 -- 172.21.15.100:0/1706190112 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc18137a60 msgr2=0x7fcc181cc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.828+0000 7fcc20486640 1 --2- 172.21.15.100:0/1706190112 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc18137a60 0x7fcc181cc590 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcc1400e9e0 tx=0x7fcc1400eea0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.828+0000 7fcc20486640 1 -- 172.21.15.100:0/1706190112 shutdown_connections 2024-09-13T11:53:09.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.828+0000 7fcc20486640 1 --2- 172.21.15.100:0/1706190112 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fcbfc03dfd0 0x7fcbfc040490 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.828+0000 7fcc20486640 1 --2- 172.21.15.100:0/1706190112 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc18138250 0x7fcc18136c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.828+0000 7fcc20486640 1 --2- 172.21.15.100:0/1706190112 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc18137e40 0x7fcc18136750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.828+0000 7fcc20486640 1 --2- 172.21.15.100:0/1706190112 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc18137a60 0x7fcc181cc590 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.828+0000 7fcc20486640 1 -- 172.21.15.100:0/1706190112 >> 172.21.15.100:0/1706190112 conn(0x7fcc180aa580 msgr2=0x7fcc180a4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:09.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.828+0000 7fcc20486640 1 -- 172.21.15.100:0/1706190112 shutdown_connections 2024-09-13T11:53:09.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.828+0000 7fcc20486640 1 -- 172.21.15.100:0/1706190112 wait complete. 2024-09-13T11:53:09.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.829+0000 7f516f8e4640 1 -- 172.21.15.100:0/7398396 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f514c03e050 msgr2=0x7f514c040510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.829+0000 7f516f8e4640 1 --2- 172.21.15.100:0/7398396 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f514c03e050 0x7f514c040510 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5164000f60 tx=0x7f5164009040 comp rx=0 tx=0).stop 2024-09-13T11:53:09.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.829+0000 7f516f8e4640 10 monclient: shutdown 2024-09-13T11:53:09.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.829+0000 7f516f8e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:09.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.829+0000 7f516f8e4640 1 -- 172.21.15.100:0/7398396 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f516813c9f0 msgr2=0x7f516809d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.829+0000 7f516f8e4640 1 --2- 172.21.15.100:0/7398396 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f516813c9f0 0x7f516809d860 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f516000c8b0 tx=0x7f516000cd70 comp rx=0 tx=0).stop 2024-09-13T11:53:09.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.829+0000 7f516f8e4640 1 -- 172.21.15.100:0/7398396 shutdown_connections 2024-09-13T11:53:09.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.829+0000 7f516f8e4640 1 --2- 172.21.15.100:0/7398396 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f514c03e050 0x7f514c040510 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.829+0000 7f516f8e4640 1 --2- 172.21.15.100:0/7398396 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f516813c9f0 0x7f516809d860 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.829+0000 7f516f8e4640 1 --2- 172.21.15.100:0/7398396 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51680a2860 0x7f51681cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.829+0000 7f516f8e4640 1 --2- 172.21.15.100:0/7398396 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f516806ea50 0x7f51681cc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.829+0000 7f516f8e4640 1 -- 172.21.15.100:0/7398396 >> 172.21.15.100:0/7398396 conn(0x7f51680a68e0 msgr2=0x7f51680a3650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:09.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.829+0000 7f516f8e4640 1 -- 172.21.15.100:0/7398396 shutdown_connections 2024-09-13T11:53:09.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.829+0000 7f516f8e4640 1 -- 172.21.15.100:0/7398396 wait complete. 2024-09-13T11:53:09.840 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640262 for osd.1 2024-09-13T11:53:09.841 DEBUG:teuthology.parallel:result is None 2024-09-13T11:53:09.841 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607558 for osd.0 2024-09-13T11:53:09.842 DEBUG:teuthology.parallel:result is None 2024-09-13T11:53:09.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.858+0000 7fdd25ff3640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-13T11:53:09.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.858+0000 7fdd25ff3640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-13T11:53:09.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.858+0000 7fdd25ff3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:53:09.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.858+0000 7fdd25ff3640 1 -- 172.21.15.100:0/170894101 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fdd20125690 con 0x7fdd2006ee30 2024-09-13T11:53:09.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.859+0000 7fdd07fff640 1 -- 172.21.15.100:0/170894101 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdd14046d90 con 0x7fdd2006ee30 2024-09-13T11:53:09.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.859+0000 7fdd07fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-13T11:53:09.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.859+0000 7fdd07fff640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:09.861 INFO:teuthology.orchestra.run.smithi100.stdout:51539607557 2024-09-13T11:53:09.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.861+0000 7fdd25ff3640 1 -- 172.21.15.100:0/170894101 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fdd0c03dff0 msgr2=0x7fdd0c0404b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.861+0000 7fdd25ff3640 1 --2- 172.21.15.100:0/170894101 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fdd0c03dff0 0x7fdd0c0404b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdd10009b40 tx=0x7fdd10009190 comp rx=0 tx=0).stop 2024-09-13T11:53:09.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.861+0000 7fdd25ff3640 10 monclient: shutdown 2024-09-13T11:53:09.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.861+0000 7fdd25ff3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:09.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.861+0000 7fdd25ff3640 1 -- 172.21.15.100:0/170894101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd2006ee30 msgr2=0x7fdd20123d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:09.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.861+0000 7fdd25ff3640 1 --2- 172.21.15.100:0/170894101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd2006ee30 0x7fdd20123d20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdd14031000 tx=0x7fdd140046c0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.861+0000 7fdd25ff3640 1 -- 172.21.15.100:0/170894101 shutdown_connections 2024-09-13T11:53:09.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.861+0000 7fdd25ff3640 1 --2- 172.21.15.100:0/170894101 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fdd0c03dff0 0x7fdd0c0404b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.861+0000 7fdd25ff3640 1 --2- 172.21.15.100:0/170894101 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd2006f790 0x7fdd20124260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.861+0000 7fdd25ff3640 1 --2- 172.21.15.100:0/170894101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd2006ee30 0x7fdd20123d20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.861+0000 7fdd25ff3640 1 --2- 172.21.15.100:0/170894101 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd2006ea50 0x7fdd201261f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:09.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.861+0000 7fdd25ff3640 1 -- 172.21.15.100:0/170894101 >> 172.21.15.100:0/170894101 conn(0x7fdd200a68e0 msgr2=0x7fdd2009da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:09.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.861+0000 7fdd25ff3640 1 -- 172.21.15.100:0/170894101 shutdown_connections 2024-09-13T11:53:09.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:09.861+0000 7fdd25ff3640 1 -- 172.21.15.100:0/170894101 wait complete. 2024-09-13T11:53:09.875 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.4 2024-09-13T11:53:09.875 DEBUG:teuthology.parallel:result is None 2024-09-13T11:53:09.875 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T11:53:09.875 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:53:10.012 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.011+0000 7f037ac8e640 10 monclient: get_monmap_and_config 2024-09-13T11:53:10.013 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.011+0000 7f037ac8e640 10 monclient: build_initial_monmap 2024-09-13T11:53:10.013 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.011+0000 7f037ac8e640 10 monclient: monmap: 2024-09-13T11:53:10.013 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:10.013 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:10.013 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:10.012075+0000 2024-09-13T11:53:10.013 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:10.012075+0000 2024-09-13T11:53:10.013 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:10.013 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:10.013 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:10.013 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:10.013 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:10.013 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:10.013 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:10.014 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:10.024 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.011+0000 7f037ac8e640 1 Processor -- start 2024-09-13T11:53:10.024 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.011+0000 7f037ac8e640 1 -- start start 2024-09-13T11:53:10.024 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.011+0000 7f037ac8e640 10 monclient: init 2024-09-13T11:53:10.024 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f037ac8e640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:10.024 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f037ac8e640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-13T11:53:10.024 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f037ac8e640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f037406ea50 0x7f03740a2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f037ac8e640 10 monclient(hunting): picked mon.noname-a con 0x7f037406ea50 addr v2:172.21.15.100:3300/0 2024-09-13T11:53:10.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f037ac8e640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f037406ee30 0x7f037406f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f037ac8e640 10 monclient(hunting): picked mon.noname-b con 0x7f037406ee30 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:10.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f037ac8e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f037406f790 0x7f03741375a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f037ac8e640 10 monclient(hunting): picked mon.noname-c con 0x7f037406f790 addr v2:172.21.15.122:3300/0 2024-09-13T11:53:10.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f037ac8e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:10.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f037ac8e640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f03740a1590 con 0x7f037406ea50 2024-09-13T11:53:10.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f037ac8e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:10.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f037ac8e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f03740a1870 con 0x7f037406f790 2024-09-13T11:53:10.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f037ac8e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:10.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f037ac8e640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f03740a1b50 con 0x7f037406ee30 2024-09-13T11:53:10.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f037ac8e640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:10.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f037ac8e640 10 monclient(hunting): authenticate will time out at 770.605463s 2024-09-13T11:53:10.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0378a2a640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f037406ea50 0x7f03740a2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0373fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f037406ee30 0x7f037406f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0378a2a640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f037406ea50 0x7f03740a2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49694/0 (socket says 172.21.15.100:49694) 2024-09-13T11:53:10.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0373fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f037406ee30 0x7f037406f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47512/0 (socket says 172.21.15.100:47512) 2024-09-13T11:53:10.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0378a2a640 1 -- 172.21.15.100:0/2994043431 learned_addr learned my addr 172.21.15.100:0/2994043431 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:10.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0378a2a640 10 monclient(hunting): get_auth_request con 0x7f037406ea50 auth_method 0 2024-09-13T11:53:10.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0378a2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:10.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0378a2a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:10.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0378a2a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:10.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0373fff640 10 monclient(hunting): get_auth_request con 0x7f037406ee30 auth_method 0 2024-09-13T11:53:10.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0373fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:10.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0373fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:10.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0373fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:10.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0373fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:10.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0373fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:10.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0373fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:10.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0378a2a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:10.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0378a2a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:10.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0378a2a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:10.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f037922b640 1 --2- 172.21.15.100:0/2994043431 >> v2:172.21.15.122:3300/0 conn(0x7f037406f790 0x7f03741375a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0378a2a640 10 monclient(hunting): handle_auth_done global_id 4362 payload 274 2024-09-13T11:53:10.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0378a2a640 1 -- 172.21.15.100:0/2994043431 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f037406ee30 msgr2=0x7f037406f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:10.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0378a2a640 1 --2- 172.21.15.100:0/2994043431 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f037406ee30 0x7f037406f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0378a2a640 1 -- 172.21.15.100:0/2994043431 >> v2:172.21.15.122:3300/0 conn(0x7f037406f790 msgr2=0x7f03741375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:10.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0378a2a640 1 --2- 172.21.15.100:0/2994043431 >> v2:172.21.15.122:3300/0 conn(0x7f037406f790 0x7f03741375a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0378a2a640 10 monclient: _finish_hunting 0 2024-09-13T11:53:10.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0378a2a640 1 monclient: found mon.noname-a 2024-09-13T11:53:10.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0378a2a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:10.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0378a2a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:10.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0378a2a640 1 -- 172.21.15.100:0/2994043431 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f037406fca0 con 0x7f037406ea50 2024-09-13T11:53:10.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.012+0000 7f0373fff640 1 --2- 172.21.15.100:0/2994043431 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f037406ee30 0x7f037406f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:10.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.013+0000 7f0378a2a640 1 --2- 172.21.15.100:0/2994043431 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f037406ea50 0x7f03740a2860 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f03740a6500 tx=0x7f036802a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0c2aa22f5587dbe server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:10.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.013+0000 7f03737fe640 1 -- 172.21.15.100:0/2994043431 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0368037070 con 0x7f037406ea50 2024-09-13T11:53:10.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.013+0000 7f03737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:10.029 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.013+0000 7f03737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:10.029 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.013+0000 7f03737fe640 10 monclient: dump: 2024-09-13T11:53:10.029 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:10.029 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:10.029 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.029 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.029 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:10.029 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:10.029 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:10.029 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:10.029 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:10.029 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:10.029 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.013+0000 7f03737fe640 10 monclient: _finish_auth 0 2024-09-13T11:53:10.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.013+0000 7f03737fe640 10 monclient: _check_auth_tickets 2024-09-13T11:53:10.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.013+0000 7f03737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:10.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.013+0000 7f03737fe640 1 -- 172.21.15.100:0/2994043431 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f036802ae00 con 0x7f037406ea50 2024-09-13T11:53:10.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.013+0000 7f037ac8e640 5 monclient: authenticate success, global_id 4362 2024-09-13T11:53:10.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.013+0000 7f037ac8e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:10.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.013+0000 7f03737fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:10.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.013+0000 7f037ac8e640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:10.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.013+0000 7f03737fe640 1 -- 172.21.15.100:0/2994043431 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0368032660 con 0x7f037406ea50 2024-09-13T11:53:10.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.013+0000 7f03737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:10.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.013+0000 7f037ac8e640 4 set_mon_vals no callback set 2024-09-13T11:53:10.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.013+0000 7f03737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:10.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.013+0000 7f03737fe640 10 monclient: dump: 2024-09-13T11:53:10.031 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:10.031 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:10.031 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.031 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.031 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:10.031 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:10.031 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:10.031 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:10.032 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:10.032 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:10.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.014+0000 7f037ac8e640 10 monclient: shutdown 2024-09-13T11:53:10.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.014+0000 7f037ac8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:10.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.014+0000 7f037ac8e640 1 -- 172.21.15.100:0/2994043431 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f037406ea50 msgr2=0x7f03740a2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:10.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.014+0000 7f037ac8e640 1 --2- 172.21.15.100:0/2994043431 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f037406ea50 0x7f03740a2860 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f03740a6500 tx=0x7f036802a0b0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.014+0000 7f037ac8e640 1 -- 172.21.15.100:0/2994043431 shutdown_connections 2024-09-13T11:53:10.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.014+0000 7f037ac8e640 1 --2- 172.21.15.100:0/2994043431 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f037406ee30 0x7f037406f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.014+0000 7f037ac8e640 1 --2- 172.21.15.100:0/2994043431 >> v2:172.21.15.122:3300/0 conn(0x7f037406f790 0x7f03741375a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.014+0000 7f037ac8e640 1 --2- 172.21.15.100:0/2994043431 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f037406ea50 0x7f03740a2860 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.033 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.014+0000 7f037ac8e640 1 -- 172.21.15.100:0/2994043431 >> 172.21.15.100:0/2994043431 conn(0x7f03740a68e0 msgr2=0x7f03740a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:10.033 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.014+0000 7f037ac8e640 1 -- 172.21.15.100:0/2994043431 shutdown_connections 2024-09-13T11:53:10.033 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.014+0000 7f037ac8e640 1 -- 172.21.15.100:0/2994043431 wait complete. 2024-09-13T11:53:10.033 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.014+0000 7f037ac8e640 10 monclient: build_initial_monmap 2024-09-13T11:53:10.033 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.014+0000 7f037ac8e640 10 monclient: monmap: 2024-09-13T11:53:10.033 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:10.034 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:10.034 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:10.034 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:10.034 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:10.034 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:10.034 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:10.034 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:10.034 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:10.034 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:10.034 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.015+0000 7f037ac8e640 1 Processor -- start 2024-09-13T11:53:10.034 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.015+0000 7f037ac8e640 1 -- start start 2024-09-13T11:53:10.034 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.015+0000 7f037ac8e640 10 monclient: init 2024-09-13T11:53:10.034 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.015+0000 7f037ac8e640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:10.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.015+0000 7f037ac8e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:53:10.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.015+0000 7f037ac8e640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f037406ea50 0x7f03741cca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037ac8e640 10 monclient(hunting): picked mon.noname-a con 0x7f037406ea50 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:10.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037ac8e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f037406ee30 0x7f03741ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037ac8e640 10 monclient(hunting): picked mon.noname-b con 0x7f037406ee30 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:10.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037ac8e640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f037406f790 0x7f0374124100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037ac8e640 10 monclient(hunting): picked mon.noname-c con 0x7f037406f790 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:10.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037ac8e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:10.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037ac8e640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f0374137c60 con 0x7f037406ea50 2024-09-13T11:53:10.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037ac8e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:10.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037ac8e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f0374137ae0 con 0x7f037406ee30 2024-09-13T11:53:10.036 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037ac8e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:10.036 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037ac8e640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f0374137de0 con 0x7f037406f790 2024-09-13T11:53:10.036 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037ac8e640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:10.036 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037ac8e640 10 monclient(hunting): authenticate will time out at 770.609080s 2024-09-13T11:53:10.036 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0378a2a640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f037406ea50 0x7f03741cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.036 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0378a2a640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f037406ea50 0x7f03741cca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49704/0 (socket says 172.21.15.100:49704) 2024-09-13T11:53:10.036 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0378a2a640 1 -- 172.21.15.100:0/3845336870 learned_addr learned my addr 172.21.15.100:0/3845336870 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:10.036 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0378a2a640 10 monclient(hunting): get_auth_request con 0x7f037406ea50 auth_method 0 2024-09-13T11:53:10.036 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0378a2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:10.036 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0378a2a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:10.036 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0378a2a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:10.037 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037922b640 1 --2- 172.21.15.100:0/3845336870 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f037406f790 0x7f0374124100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.037 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0378a2a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:10.037 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0378a2a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:10.037 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0378a2a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:10.037 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037922b640 10 monclient(hunting): get_auth_request con 0x7f037406f790 auth_method 0 2024-09-13T11:53:10.037 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037922b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:10.037 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037922b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:10.037 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037922b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:10.037 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037922b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:10.037 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037922b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:10.037 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037922b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:10.037 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0378a2a640 10 monclient(hunting): handle_auth_done global_id 4365 payload 824 2024-09-13T11:53:10.038 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0378a2a640 1 -- 172.21.15.100:0/3845336870 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f037406f790 msgr2=0x7f0374124100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:10.038 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0378a2a640 1 --2- 172.21.15.100:0/3845336870 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f037406f790 0x7f0374124100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.038 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0373fff640 1 --2- 172.21.15.100:0/3845336870 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f037406ee30 0x7f03741ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.038 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0378a2a640 1 -- 172.21.15.100:0/3845336870 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f037406ee30 msgr2=0x7f03741ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:10.038 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0378a2a640 1 --2- 172.21.15.100:0/3845336870 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f037406ee30 0x7f03741ccf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.038 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0378a2a640 10 monclient: _finish_hunting 0 2024-09-13T11:53:10.038 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0378a2a640 1 monclient: found mon.noname-a 2024-09-13T11:53:10.038 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0378a2a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:10.038 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0378a2a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:10.038 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0378a2a640 1 -- 172.21.15.100:0/3845336870 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0374125a30 con 0x7f037406ea50 2024-09-13T11:53:10.038 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0373fff640 10 monclient: get_auth_request con 0x7f037406ee30 auth_method 0 2024-09-13T11:53:10.038 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0373fff640 1 --2- 172.21.15.100:0/3845336870 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f037406ee30 0x7f03741ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:53:10.039 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0378a2a640 1 --2- 172.21.15.100:0/3845336870 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f037406ea50 0x7f03741cca40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f036802a0f0 tx=0x7f03680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:10.039 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f03717fa640 1 -- 172.21.15.100:0/3845336870 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0368041070 con 0x7f037406ea50 2024-09-13T11:53:10.039 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f03717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:10.039 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f03717fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:10.039 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f03717fa640 10 monclient: dump: 2024-09-13T11:53:10.039 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:10.039 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:10.039 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.039 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.039 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:10.039 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:10.039 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:10.039 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:10.040 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:10.040 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:10.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f03717fa640 10 monclient: _finish_auth 0 2024-09-13T11:53:10.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f03717fa640 10 monclient: _check_auth_tickets 2024-09-13T11:53:10.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f03717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:10.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f03717fa640 1 -- 172.21.15.100:0/3845336870 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f036803c460 con 0x7f037406ea50 2024-09-13T11:53:10.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037ac8e640 5 monclient: authenticate success, global_id 4365 2024-09-13T11:53:10.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037ac8e640 10 monclient: _renew_subs 2024-09-13T11:53:10.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037ac8e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:10.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037ac8e640 1 -- 172.21.15.100:0/3845336870 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0374126590 con 0x7f037406ea50 2024-09-13T11:53:10.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f03717fa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:10.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f03717fa640 1 -- 172.21.15.100:0/3845336870 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0368037040 con 0x7f037406ea50 2024-09-13T11:53:10.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f03717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:10.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f03717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:10.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f03717fa640 10 monclient: dump: 2024-09-13T11:53:10.041 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:10.041 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:10.041 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.041 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.041 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:10.041 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:10.041 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:10.041 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:10.041 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:10.041 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:10.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f0372ffd640 4 set_mon_vals no callback set 2024-09-13T11:53:10.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037ac8e640 10 monclient: _renew_subs 2024-09-13T11:53:10.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037ac8e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:10.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.016+0000 7f037ac8e640 1 -- 172.21.15.100:0/3845336870 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03741274f0 con 0x7f037406ea50 2024-09-13T11:53:10.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.017+0000 7f03717fa640 1 -- 172.21.15.100:0/3845336870 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f036803a070 con 0x7f037406ea50 2024-09-13T11:53:10.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.017+0000 7f037ac8e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:10.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.017+0000 7f037ac8e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:10.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.017+0000 7f037ac8e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:10.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.017+0000 7f037ac8e640 1 -- 172.21.15.100:0/3845336870 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03741381f0 con 0x7f037406ea50 2024-09-13T11:53:10.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.017+0000 7f03717fa640 1 --2- 172.21.15.100:0/3845336870 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f035803dfa0 0x7f0358040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.017+0000 7f03717fa640 1 -- 172.21.15.100:0/3845336870 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7f03680728b0 con 0x7f037406ea50 2024-09-13T11:53:10.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.018+0000 7f0373fff640 1 --2- 172.21.15.100:0/3845336870 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f035803dfa0 0x7f0358040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.018+0000 7f0373fff640 10 monclient: get_auth_request con 0x7f035803dfa0 auth_method 0 2024-09-13T11:53:10.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.018+0000 7f0373fff640 1 --2- 172.21.15.100:0/3845336870 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f035803dfa0 0x7f0358040460 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0374126520 tx=0x7f0360008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:10.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.022+0000 7f03717fa640 1 -- 172.21.15.100:0/3845336870 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f0368030d50 con 0x7f037406ea50 2024-09-13T11:53:10.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.022+0000 7f03717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:10.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.022+0000 7f03717fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:10.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.191+0000 7f037ac8e640 1 -- 172.21.15.100:0/3845336870 --> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0374127c40 con 0x7f035803dfa0 2024-09-13T11:53:10.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.193+0000 7f03717fa640 1 -- 172.21.15.100:0/3845336870 <== mgr.4114 v2:172.21.15.122:6824/441789830 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34178 (secure 0 0 0) 0x7f0360010930 con 0x7f035803dfa0 2024-09-13T11:53:10.195 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:53:10.195 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-13T11:53:10.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.194+0000 7f037ac8e640 1 -- 172.21.15.100:0/3845336870 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f035803dfa0 msgr2=0x7f0358040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:10.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.194+0000 7f037ac8e640 1 --2- 172.21.15.100:0/3845336870 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f035803dfa0 0x7f0358040460 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0374126520 tx=0x7f0360008040 comp rx=0 tx=0).stop 2024-09-13T11:53:10.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.194+0000 7f037ac8e640 10 monclient: shutdown 2024-09-13T11:53:10.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.194+0000 7f037ac8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:10.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.194+0000 7f037ac8e640 1 -- 172.21.15.100:0/3845336870 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f037406ea50 msgr2=0x7f03741cca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:10.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.194+0000 7f037ac8e640 1 --2- 172.21.15.100:0/3845336870 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f037406ea50 0x7f03741cca40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f036802a0f0 tx=0x7f03680046c0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.194+0000 7f037ac8e640 1 -- 172.21.15.100:0/3845336870 shutdown_connections 2024-09-13T11:53:10.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.194+0000 7f037ac8e640 1 --2- 172.21.15.100:0/3845336870 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f035803dfa0 0x7f0358040460 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.194+0000 7f037ac8e640 1 --2- 172.21.15.100:0/3845336870 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f037406f790 0x7f0374124100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.194+0000 7f037ac8e640 1 --2- 172.21.15.100:0/3845336870 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f037406ee30 0x7f03741ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.194+0000 7f037ac8e640 1 --2- 172.21.15.100:0/3845336870 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f037406ea50 0x7f03741cca40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.194+0000 7f037ac8e640 1 -- 172.21.15.100:0/3845336870 >> 172.21.15.100:0/3845336870 conn(0x7f03740a68e0 msgr2=0x7f03740a0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:10.199 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.194+0000 7f037ac8e640 1 -- 172.21.15.100:0/3845336870 shutdown_connections 2024-09-13T11:53:10.199 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.194+0000 7f037ac8e640 1 -- 172.21.15.100:0/3845336870 wait complete. 2024-09-13T11:53:10.207 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-13T11:53:09.211951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163332,"kb_used_data":1136,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067708,"statfs":{"total":579820584960,"available":579653332992,"internally_reserved":0,"allocated":1163264,"data_stored":1397488,"data_compressed":59246,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.958072"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.266756+0000","last_change":"2024-09-13T11:53:06.266756+0000","last_active":"2024-09-13T11:53:06.266756+0000","last_peered":"2024-09-13T11:53:06.266756+0000","last_clean":"2024-09-13T11:53:06.266756+0000","last_became_active":"2024-09-13T11:53:04.566167+0000","last_became_peered":"2024-09-13T11:53:04.566167+0000","last_unstale":"2024-09-13T11:53:06.266756+0000","last_undegraded":"2024-09-13T11:53:06.266756+0000","last_fullsized":"2024-09-13T11:53:06.266756+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:41.983816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084363899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.266077+0000","last_change":"2024-09-13T11:53:06.266346+0000","last_active":"2024-09-13T11:53:06.266077+0000","last_peered":"2024-09-13T11:53:06.266077+0000","last_clean":"2024-09-13T11:53:06.266077+0000","last_became_active":"2024-09-13T11:53:04.258975+0000","last_became_peered":"2024-09-13T11:53:04.258975+0000","last_unstale":"2024-09-13T11:53:06.266077+0000","last_undegraded":"2024-09-13T11:53:06.266077+0000","last_fullsized":"2024-09-13T11:53:06.266077+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:36.773393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055464299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.266446+0000","last_change":"2024-09-13T11:53:06.266446+0000","last_active":"2024-09-13T11:53:06.266446+0000","last_peered":"2024-09-13T11:53:06.266446+0000","last_clean":"2024-09-13T11:53:06.266446+0000","last_became_active":"2024-09-13T11:53:04.262475+0000","last_became_peered":"2024-09-13T11:53:04.262475+0000","last_unstale":"2024-09-13T11:53:06.266446+0000","last_undegraded":"2024-09-13T11:53:06.266446+0000","last_fullsized":"2024-09-13T11:53:06.266446+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:24:12.639511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055213099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.266124+0000","last_change":"2024-09-13T11:53:06.266390+0000","last_active":"2024-09-13T11:53:06.266124+0000","last_peered":"2024-09-13T11:53:06.266124+0000","last_clean":"2024-09-13T11:53:06.266124+0000","last_became_active":"2024-09-13T11:53:04.262392+0000","last_became_peered":"2024-09-13T11:53:04.262392+0000","last_unstale":"2024-09-13T11:53:06.266124+0000","last_undegraded":"2024-09-13T11:53:06.266124+0000","last_fullsized":"2024-09-13T11:53:06.266124+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:45.666214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057483700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.272635+0000","last_change":"2024-09-13T11:53:06.272635+0000","last_active":"2024-09-13T11:53:06.272635+0000","last_peered":"2024-09-13T11:53:06.272635+0000","last_clean":"2024-09-13T11:53:06.272635+0000","last_became_active":"2024-09-13T11:53:04.264659+0000","last_became_peered":"2024-09-13T11:53:04.264659+0000","last_unstale":"2024-09-13T11:53:06.272635+0000","last_undegraded":"2024-09-13T11:53:06.272635+0000","last_fullsized":"2024-09-13T11:53:06.272635+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:53:49.774126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010464509999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.265979+0000","last_change":"2024-09-13T11:53:06.266298+0000","last_active":"2024-09-13T11:53:06.265979+0000","last_peered":"2024-09-13T11:53:06.265979+0000","last_clean":"2024-09-13T11:53:06.265979+0000","last_became_active":"2024-09-13T11:53:04.270146+0000","last_became_peered":"2024-09-13T11:53:04.270146+0000","last_unstale":"2024-09-13T11:53:06.265979+0000","last_undegraded":"2024-09-13T11:53:06.265979+0000","last_fullsized":"2024-09-13T11:53:06.265979+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:38.439658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061806000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.266345+0000","last_change":"2024-09-13T11:53:06.266345+0000","last_active":"2024-09-13T11:53:06.266345+0000","last_peered":"2024-09-13T11:53:06.266345+0000","last_clean":"2024-09-13T11:53:06.266345+0000","last_became_active":"2024-09-13T11:53:04.262197+0000","last_became_peered":"2024-09-13T11:53:04.262197+0000","last_unstale":"2024-09-13T11:53:06.266345+0000","last_undegraded":"2024-09-13T11:53:06.266345+0000","last_fullsized":"2024-09-13T11:53:06.266345+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:05:51.189944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044149899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":21,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.271834+0000","last_change":"2024-09-13T11:53:06.272071+0000","last_active":"2024-09-13T11:53:06.271834+0000","last_peered":"2024-09-13T11:53:06.271834+0000","last_clean":"2024-09-13T11:53:06.271834+0000","last_became_active":"2024-09-13T11:53:04.270138+0000","last_became_peered":"2024-09-13T11:53:04.270138+0000","last_unstale":"2024-09-13T11:53:06.271834+0000","last_undegraded":"2024-09-13T11:53:06.271834+0000","last_fullsized":"2024-09-13T11:53:06.271834+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:17.090753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047405,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":111,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.265801+0000","last_change":"2024-09-13T11:52:57.706946+0000","last_active":"2024-09-13T11:53:06.265801+0000","last_peered":"2024-09-13T11:53:06.265801+0000","last_clean":"2024-09-13T11:53:06.265801+0000","last_became_active":"2024-09-13T11:52:57.705894+0000","last_became_peered":"2024-09-13T11:52:57.705894+0000","last_unstale":"2024-09-13T11:53:06.265801+0000","last_undegraded":"2024-09-13T11:53:06.265801+0000","last_fullsized":"2024-09-13T11:53:06.265801+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:52:56.225375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:52:56.225375+0000","last_clean_scrub_stamp":"2024-09-13T11:52:56.225375+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:05.299261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59246,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":172032,"data_stored":56599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607557,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":233472,"data_stored":585527,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607558,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":585527,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":176128,"data_stored":56618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":172032,"data_stored":56599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640262,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":176128,"data_stored":56618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:53:10.208 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:53:10.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.302+0000 7ff5f2abf640 10 monclient: get_monmap_and_config 2024-09-13T11:53:10.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.302+0000 7ff5f2abf640 10 monclient: build_initial_monmap 2024-09-13T11:53:10.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.302+0000 7ff5f2abf640 10 monclient: monmap: 2024-09-13T11:53:10.305 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:10.305 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:10.305 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:10.304433+0000 2024-09-13T11:53:10.305 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:10.304433+0000 2024-09-13T11:53:10.305 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:10.305 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:10.305 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:10.305 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:10.305 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:10.305 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:10.306 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:10.306 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:10.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.303+0000 7ff5f2abf640 1 Processor -- start 2024-09-13T11:53:10.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.303+0000 7ff5f2abf640 1 -- start start 2024-09-13T11:53:10.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.303+0000 7ff5f2abf640 10 monclient: init 2024-09-13T11:53:10.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.303+0000 7ff5f2abf640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:10.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.303+0000 7ff5f2abf640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-13T11:53:10.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.303+0000 7ff5f2abf640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff5ec0a3440 0x7ff5ec0a3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.303+0000 7ff5f2abf640 10 monclient(hunting): picked mon.noname-c con 0x7ff5ec0a3440 addr v2:172.21.15.122:3300/0 2024-09-13T11:53:10.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.303+0000 7ff5f2abf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff5ec0a28a0 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:10.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.303+0000 7ff5f2abf640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5ec0a4ff0 0x7ff5ec13f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.303+0000 7ff5f2abf640 10 monclient(hunting): picked mon.noname-b con 0x7ff5ec0a4ff0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:10.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.303+0000 7ff5f2abf640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5ec0a3d60 con 0x7ff5ec0a28a0 2024-09-13T11:53:10.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.303+0000 7ff5f2abf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:10.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.303+0000 7ff5f2abf640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7ff5ec0a9d20 con 0x7ff5ec0a3440 2024-09-13T11:53:10.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.303+0000 7ff5f2abf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:10.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.303+0000 7ff5f2abf640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7ff5ec0a67b0 con 0x7ff5ec0a4ff0 2024-09-13T11:53:10.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.303+0000 7ff5f2abf640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:10.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.303+0000 7ff5f2abf640 10 monclient(hunting): authenticate will time out at 770.897852s 2024-09-13T11:53:10.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.303+0000 7ff5f105c640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5ec0a4ff0 0x7ff5ec13f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.303+0000 7ff5ebfff640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7ff5ec0a28a0 0x7ff5ec0a48e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:58036/0 (socket says 172.21.15.100:58036) 2024-09-13T11:53:10.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.303+0000 7ff5ebfff640 1 -- 172.21.15.100:0/3064566235 learned_addr learned my addr 172.21.15.100:0/3064566235 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:53:10.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.303+0000 7ff5f105c640 10 monclient(hunting): get_auth_request con 0x7ff5ec0a4ff0 auth_method 0 2024-09-13T11:53:10.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.304+0000 7ff5f105c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:10.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.304+0000 7ff5f105c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:10.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.304+0000 7ff5f105c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:10.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.304+0000 7ff5f105c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:10.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.304+0000 7ff5f105c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:10.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.304+0000 7ff5f105c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:10.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.304+0000 7ff5f085b640 1 --2- 172.21.15.100:0/3064566235 >> v2:172.21.15.122:3300/0 conn(0x7ff5ec0a3440 0x7ff5ec0a3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.304+0000 7ff5f085b640 10 monclient(hunting): get_auth_request con 0x7ff5ec0a3440 auth_method 0 2024-09-13T11:53:10.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.304+0000 7ff5f085b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:10.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.304+0000 7ff5f085b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:10.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.304+0000 7ff5f085b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:10.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.304+0000 7ff5f105c640 10 monclient(hunting): handle_auth_done global_id 4334 payload 274 2024-09-13T11:53:10.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.304+0000 7ff5f105c640 1 -- 172.21.15.100:0/3064566235 >> v2:172.21.15.122:3300/0 conn(0x7ff5ec0a3440 msgr2=0x7ff5ec0a3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:10.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.304+0000 7ff5f105c640 1 --2- 172.21.15.100:0/3064566235 >> v2:172.21.15.122:3300/0 conn(0x7ff5ec0a3440 0x7ff5ec0a3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.304+0000 7ff5eb7fe640 1 -- 172.21.15.100:0/3064566235 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3973980543 0 0) 0x7ff5d8001110 con 0x7ff5ec0a28a0 2024-09-13T11:53:10.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.304+0000 7ff5f105c640 1 -- 172.21.15.100:0/3064566235 >> v1:172.21.15.100:6789/0 conn(0x7ff5ec0a28a0 legacy=0x7ff5ec0a48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:10.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.304+0000 7ff5f105c640 10 monclient: _finish_hunting 0 2024-09-13T11:53:10.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.304+0000 7ff5f105c640 1 monclient: found mon.noname-b 2024-09-13T11:53:10.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.304+0000 7ff5f105c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:10.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.304+0000 7ff5f105c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:53:10.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.304+0000 7ff5f105c640 1 -- 172.21.15.100:0/3064566235 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5ec0a1620 con 0x7ff5ec0a4ff0 2024-09-13T11:53:10.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.304+0000 7ff5f085b640 1 --2- 172.21.15.100:0/3064566235 >> v2:172.21.15.122:3300/0 conn(0x7ff5ec0a3440 0x7ff5ec0a3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:53:10.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.304+0000 7ff5eb7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:53:10.317 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.304+0000 7ff5f105c640 1 --2- 172.21.15.100:0/3064566235 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5ec0a4ff0 0x7ff5ec13f900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4009d90 tx=0x7ff5e402a530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4d07ffe55c7b643 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:10.317 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5eb7fe640 1 -- 172.21.15.100:0/3064566235 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e4007740 con 0x7ff5ec0a4ff0 2024-09-13T11:53:10.317 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:10.317 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5eb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:10.317 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5eb7fe640 10 monclient: dump: 2024-09-13T11:53:10.317 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:10.317 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:10.317 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.317 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.317 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:10.317 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:10.317 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:10.317 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:10.317 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:10.318 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:10.318 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5eb7fe640 10 monclient: _finish_auth 0 2024-09-13T11:53:10.318 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5eb7fe640 10 monclient: _check_auth_tickets 2024-09-13T11:53:10.318 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:10.318 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5f2abf640 5 monclient: authenticate success, global_id 4334 2024-09-13T11:53:10.318 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5eb7fe640 1 -- 172.21.15.100:0/3064566235 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5e40078e0 con 0x7ff5ec0a4ff0 2024-09-13T11:53:10.318 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5eb7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:10.318 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5f2abf640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:10.318 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5f2abf640 4 set_mon_vals no callback set 2024-09-13T11:53:10.318 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5eb7fe640 1 -- 172.21.15.100:0/3064566235 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e4007cb0 con 0x7ff5ec0a4ff0 2024-09-13T11:53:10.318 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:10.318 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5eb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:10.318 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5eb7fe640 10 monclient: dump: 2024-09-13T11:53:10.319 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:10.319 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:10.319 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.319 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.319 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:10.319 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:10.319 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:10.319 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:10.319 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:10.319 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:10.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5f2abf640 10 monclient: shutdown 2024-09-13T11:53:10.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5f2abf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:10.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5f2abf640 1 -- 172.21.15.100:0/3064566235 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5ec0a4ff0 msgr2=0x7ff5ec13f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:10.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5f2abf640 1 --2- 172.21.15.100:0/3064566235 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5ec0a4ff0 0x7ff5ec13f900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4009d90 tx=0x7ff5e402a530 comp rx=0 tx=0).stop 2024-09-13T11:53:10.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5f2abf640 1 -- 172.21.15.100:0/3064566235 shutdown_connections 2024-09-13T11:53:10.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5f2abf640 1 --2- 172.21.15.100:0/3064566235 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5ec0a4ff0 0x7ff5ec13f900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5f2abf640 1 --2- 172.21.15.100:0/3064566235 >> v2:172.21.15.122:3300/0 conn(0x7ff5ec0a3440 0x7ff5ec0a3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5f2abf640 1 -- 172.21.15.100:0/3064566235 >> 172.21.15.100:0/3064566235 conn(0x7ff5ec0a6e00 msgr2=0x7ff5ec0a7210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:10.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5f2abf640 1 -- 172.21.15.100:0/3064566235 shutdown_connections 2024-09-13T11:53:10.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.305+0000 7ff5f2abf640 1 -- 172.21.15.100:0/3064566235 wait complete. 2024-09-13T11:53:10.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.306+0000 7ff5f2abf640 10 monclient: build_initial_monmap 2024-09-13T11:53:10.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.306+0000 7ff5f2abf640 10 monclient: monmap: 2024-09-13T11:53:10.320 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:10.320 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:10.320 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:10.320 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:10.320 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:10.320 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:10.321 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:10.321 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:10.321 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:10.321 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:10.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.306+0000 7ff5f2abf640 1 Processor -- start 2024-09-13T11:53:10.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.306+0000 7ff5f2abf640 1 -- start start 2024-09-13T11:53:10.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.306+0000 7ff5f2abf640 10 monclient: init 2024-09-13T11:53:10.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f2abf640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:10.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f2abf640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:53:10.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f2abf640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5ec0a28a0 0x7ff5ec1ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f2abf640 10 monclient(hunting): picked mon.noname-c con 0x7ff5ec0a28a0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:10.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f2abf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5ec0a3440 0x7ff5ec1c07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f2abf640 10 monclient(hunting): picked mon.noname-b con 0x7ff5ec0a3440 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:10.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f2abf640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5ec0a4ff0 0x7ff5ec1c0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f2abf640 10 monclient(hunting): picked mon.noname-a con 0x7ff5ec0a4ff0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:10.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f2abf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:10.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f2abf640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7ff5ec0a2e50 con 0x7ff5ec0a4ff0 2024-09-13T11:53:10.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f2abf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:10.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f2abf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff5ec0a2cd0 con 0x7ff5ec0a3440 2024-09-13T11:53:10.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f2abf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:10.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f2abf640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7ff5ec0a9760 con 0x7ff5ec0a28a0 2024-09-13T11:53:10.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f2abf640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:10.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f2abf640 10 monclient(hunting): authenticate will time out at 770.901332s 2024-09-13T11:53:10.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f085b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5ec0a28a0 0x7ff5ec1ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f085b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5ec0a28a0 0x7ff5ec1ccab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47542/0 (socket says 172.21.15.100:47542) 2024-09-13T11:53:10.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f085b640 1 -- 172.21.15.100:0/2205729774 learned_addr learned my addr 172.21.15.100:0/2205729774 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:10.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f085b640 10 monclient(hunting): get_auth_request con 0x7ff5ec0a28a0 auth_method 0 2024-09-13T11:53:10.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f085b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:10.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f085b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:10.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f085b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:10.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f105c640 1 --2- 172.21.15.100:0/2205729774 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5ec0a4ff0 0x7ff5ec1c0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f105c640 10 monclient(hunting): get_auth_request con 0x7ff5ec0a4ff0 auth_method 0 2024-09-13T11:53:10.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f105c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:10.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f105c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:10.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f105c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:10.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f085b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:10.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f085b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:10.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f085b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:10.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f105c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:10.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f105c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:10.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5ebfff640 1 --2- 172.21.15.100:0/2205729774 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5ec0a3440 0x7ff5ec1c07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f105c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:10.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5ebfff640 10 monclient(hunting): get_auth_request con 0x7ff5ec0a3440 auth_method 0 2024-09-13T11:53:10.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:10.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:10.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:10.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:10.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:10.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:10.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f085b640 10 monclient(hunting): handle_auth_done global_id 4337 payload 824 2024-09-13T11:53:10.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f085b640 1 -- 172.21.15.100:0/2205729774 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5ec0a3440 msgr2=0x7ff5ec1c07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:10.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f085b640 1 --2- 172.21.15.100:0/2205729774 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5ec0a3440 0x7ff5ec1c07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f085b640 1 -- 172.21.15.100:0/2205729774 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5ec0a4ff0 msgr2=0x7ff5ec1c0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:10.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f085b640 1 --2- 172.21.15.100:0/2205729774 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5ec0a4ff0 0x7ff5ec1c0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f085b640 10 monclient: _finish_hunting 0 2024-09-13T11:53:10.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f085b640 1 monclient: found mon.noname-c 2024-09-13T11:53:10.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f085b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:10.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f085b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:53:10.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f085b640 1 -- 172.21.15.100:0/2205729774 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5ec0a10d0 con 0x7ff5ec0a28a0 2024-09-13T11:53:10.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f105c640 1 --2- 172.21.15.100:0/2205729774 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5ec0a4ff0 0x7ff5ec1c0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:10.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5f085b640 1 --2- 172.21.15.100:0/2205729774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5ec0a28a0 0x7ff5ec1ccab0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff5e000e730 tx=0x7ff5e000ebf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:10.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5e97fa640 1 -- 172.21.15.100:0/2205729774 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e000f040 con 0x7ff5ec0a28a0 2024-09-13T11:53:10.326 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.307+0000 7ff5e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:10.326 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.308+0000 7ff5e97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:10.326 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.308+0000 7ff5e97fa640 10 monclient: dump: 2024-09-13T11:53:10.326 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:10.326 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:10.326 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.326 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.326 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:10.326 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:10.326 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:10.326 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:10.326 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:10.326 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:10.326 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.308+0000 7ff5e97fa640 10 monclient: _finish_auth 0 2024-09-13T11:53:10.327 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.308+0000 7ff5e97fa640 10 monclient: _check_auth_tickets 2024-09-13T11:53:10.327 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.308+0000 7ff5e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:10.327 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.308+0000 7ff5e97fa640 1 -- 172.21.15.100:0/2205729774 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5e0004230 con 0x7ff5ec0a28a0 2024-09-13T11:53:10.327 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.308+0000 7ff5f2abf640 5 monclient: authenticate success, global_id 4337 2024-09-13T11:53:10.327 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.308+0000 7ff5e97fa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:10.327 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.308+0000 7ff5e97fa640 1 -- 172.21.15.100:0/2205729774 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e000ee40 con 0x7ff5ec0a28a0 2024-09-13T11:53:10.327 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.308+0000 7ff5e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:10.327 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.308+0000 7ff5e97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:10.327 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.308+0000 7ff5e97fa640 10 monclient: dump: 2024-09-13T11:53:10.327 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:10.327 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:10.327 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.327 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.327 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:10.328 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:10.328 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:10.328 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:10.328 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:10.328 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:10.328 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.308+0000 7ff5f2abf640 10 monclient: _renew_subs 2024-09-13T11:53:10.328 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.308+0000 7ff5f2abf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:10.328 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.308+0000 7ff5f2abf640 1 -- 172.21.15.100:0/2205729774 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5ec1aded0 con 0x7ff5ec0a28a0 2024-09-13T11:53:10.328 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.308+0000 7ff5eaffd640 4 set_mon_vals no callback set 2024-09-13T11:53:10.328 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.308+0000 7ff5f2abf640 10 monclient: _renew_subs 2024-09-13T11:53:10.328 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.308+0000 7ff5f2abf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:10.328 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.308+0000 7ff5f2abf640 1 -- 172.21.15.100:0/2205729774 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5ec0706b0 con 0x7ff5ec0a28a0 2024-09-13T11:53:10.328 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.308+0000 7ff5e97fa640 1 -- 172.21.15.100:0/2205729774 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7ff5e000c6d0 con 0x7ff5ec0a28a0 2024-09-13T11:53:10.329 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.308+0000 7ff5f2abf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:10.329 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.308+0000 7ff5f2abf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:10.329 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.308+0000 7ff5f2abf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:10.329 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.308+0000 7ff5f2abf640 1 -- 172.21.15.100:0/2205729774 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5ec09f930 con 0x7ff5ec0a28a0 2024-09-13T11:53:10.329 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.309+0000 7ff5e97fa640 1 --2- 172.21.15.100:0/2205729774 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff5d003dfa0 0x7ff5d0040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.329 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.309+0000 7ff5e97fa640 1 -- 172.21.15.100:0/2205729774 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7ff5e00523f0 con 0x7ff5ec0a28a0 2024-09-13T11:53:10.329 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.309+0000 7ff5ebfff640 1 --2- 172.21.15.100:0/2205729774 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff5d003dfa0 0x7ff5d0040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.329 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.309+0000 7ff5ebfff640 10 monclient: get_auth_request con 0x7ff5d003dfa0 auth_method 0 2024-09-13T11:53:10.329 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.309+0000 7ff5ebfff640 1 --2- 172.21.15.100:0/2205729774 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff5d003dfa0 0x7ff5d0040460 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec1abb40 tx=0x7ff5d8009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:10.330 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.313+0000 7ff5e97fa640 1 -- 172.21.15.100:0/2205729774 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7ff5ec09f930 con 0x7ff5ec0a28a0 2024-09-13T11:53:10.330 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.313+0000 7ff5e97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:10.330 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.313+0000 7ff5e97fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:10.485 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.483+0000 7ff5f2abf640 1 -- 172.21.15.100:0/2205729774 --> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff5ec074f70 con 0x7ff5d003dfa0 2024-09-13T11:53:10.487 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.484+0000 7ff5e97fa640 1 -- 172.21.15.100:0/2205729774 <== mgr.4114 v2:172.21.15.122:6824/441789830 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34178 (secure 0 0 0) 0x7ff5ec074f70 con 0x7ff5d003dfa0 2024-09-13T11:53:10.488 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:53:10.489 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-13T11:53:10.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.486+0000 7ff5f2abf640 1 -- 172.21.15.100:0/2205729774 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff5d003dfa0 msgr2=0x7ff5d0040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:10.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.486+0000 7ff5f2abf640 1 --2- 172.21.15.100:0/2205729774 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff5d003dfa0 0x7ff5d0040460 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec1abb40 tx=0x7ff5d8009040 comp rx=0 tx=0).stop 2024-09-13T11:53:10.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.486+0000 7ff5f2abf640 10 monclient: shutdown 2024-09-13T11:53:10.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.486+0000 7ff5f2abf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:10.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.486+0000 7ff5f2abf640 1 -- 172.21.15.100:0/2205729774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5ec0a28a0 msgr2=0x7ff5ec1ccab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:10.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.486+0000 7ff5f2abf640 1 --2- 172.21.15.100:0/2205729774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5ec0a28a0 0x7ff5ec1ccab0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff5e000e730 tx=0x7ff5e000ebf0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.486+0000 7ff5f2abf640 1 -- 172.21.15.100:0/2205729774 shutdown_connections 2024-09-13T11:53:10.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.486+0000 7ff5f2abf640 1 --2- 172.21.15.100:0/2205729774 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff5d003dfa0 0x7ff5d0040460 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.486+0000 7ff5f2abf640 1 --2- 172.21.15.100:0/2205729774 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff5ec0a4ff0 0x7ff5ec1c0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.486+0000 7ff5f2abf640 1 --2- 172.21.15.100:0/2205729774 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5ec0a3440 0x7ff5ec1c07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.486+0000 7ff5f2abf640 1 --2- 172.21.15.100:0/2205729774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5ec0a28a0 0x7ff5ec1ccab0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.486+0000 7ff5f2abf640 1 -- 172.21.15.100:0/2205729774 >> 172.21.15.100:0/2205729774 conn(0x7ff5ec0a6e00 msgr2=0x7ff5ec09f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:10.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.486+0000 7ff5f2abf640 1 -- 172.21.15.100:0/2205729774 shutdown_connections 2024-09-13T11:53:10.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.486+0000 7ff5f2abf640 1 -- 172.21.15.100:0/2205729774 wait complete. 2024-09-13T11:53:10.499 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-13T11:53:09.211951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163332,"kb_used_data":1136,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067708,"statfs":{"total":579820584960,"available":579653332992,"internally_reserved":0,"allocated":1163264,"data_stored":1397488,"data_compressed":59246,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.958072"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.266756+0000","last_change":"2024-09-13T11:53:06.266756+0000","last_active":"2024-09-13T11:53:06.266756+0000","last_peered":"2024-09-13T11:53:06.266756+0000","last_clean":"2024-09-13T11:53:06.266756+0000","last_became_active":"2024-09-13T11:53:04.566167+0000","last_became_peered":"2024-09-13T11:53:04.566167+0000","last_unstale":"2024-09-13T11:53:06.266756+0000","last_undegraded":"2024-09-13T11:53:06.266756+0000","last_fullsized":"2024-09-13T11:53:06.266756+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:41.983816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084363899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.266077+0000","last_change":"2024-09-13T11:53:06.266346+0000","last_active":"2024-09-13T11:53:06.266077+0000","last_peered":"2024-09-13T11:53:06.266077+0000","last_clean":"2024-09-13T11:53:06.266077+0000","last_became_active":"2024-09-13T11:53:04.258975+0000","last_became_peered":"2024-09-13T11:53:04.258975+0000","last_unstale":"2024-09-13T11:53:06.266077+0000","last_undegraded":"2024-09-13T11:53:06.266077+0000","last_fullsized":"2024-09-13T11:53:06.266077+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:36.773393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055464299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.266446+0000","last_change":"2024-09-13T11:53:06.266446+0000","last_active":"2024-09-13T11:53:06.266446+0000","last_peered":"2024-09-13T11:53:06.266446+0000","last_clean":"2024-09-13T11:53:06.266446+0000","last_became_active":"2024-09-13T11:53:04.262475+0000","last_became_peered":"2024-09-13T11:53:04.262475+0000","last_unstale":"2024-09-13T11:53:06.266446+0000","last_undegraded":"2024-09-13T11:53:06.266446+0000","last_fullsized":"2024-09-13T11:53:06.266446+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:24:12.639511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055213099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.266124+0000","last_change":"2024-09-13T11:53:06.266390+0000","last_active":"2024-09-13T11:53:06.266124+0000","last_peered":"2024-09-13T11:53:06.266124+0000","last_clean":"2024-09-13T11:53:06.266124+0000","last_became_active":"2024-09-13T11:53:04.262392+0000","last_became_peered":"2024-09-13T11:53:04.262392+0000","last_unstale":"2024-09-13T11:53:06.266124+0000","last_undegraded":"2024-09-13T11:53:06.266124+0000","last_fullsized":"2024-09-13T11:53:06.266124+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:45.666214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057483700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.272635+0000","last_change":"2024-09-13T11:53:06.272635+0000","last_active":"2024-09-13T11:53:06.272635+0000","last_peered":"2024-09-13T11:53:06.272635+0000","last_clean":"2024-09-13T11:53:06.272635+0000","last_became_active":"2024-09-13T11:53:04.264659+0000","last_became_peered":"2024-09-13T11:53:04.264659+0000","last_unstale":"2024-09-13T11:53:06.272635+0000","last_undegraded":"2024-09-13T11:53:06.272635+0000","last_fullsized":"2024-09-13T11:53:06.272635+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:53:49.774126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010464509999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.265979+0000","last_change":"2024-09-13T11:53:06.266298+0000","last_active":"2024-09-13T11:53:06.265979+0000","last_peered":"2024-09-13T11:53:06.265979+0000","last_clean":"2024-09-13T11:53:06.265979+0000","last_became_active":"2024-09-13T11:53:04.270146+0000","last_became_peered":"2024-09-13T11:53:04.270146+0000","last_unstale":"2024-09-13T11:53:06.265979+0000","last_undegraded":"2024-09-13T11:53:06.265979+0000","last_fullsized":"2024-09-13T11:53:06.265979+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:38.439658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061806000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.266345+0000","last_change":"2024-09-13T11:53:06.266345+0000","last_active":"2024-09-13T11:53:06.266345+0000","last_peered":"2024-09-13T11:53:06.266345+0000","last_clean":"2024-09-13T11:53:06.266345+0000","last_became_active":"2024-09-13T11:53:04.262197+0000","last_became_peered":"2024-09-13T11:53:04.262197+0000","last_unstale":"2024-09-13T11:53:06.266345+0000","last_undegraded":"2024-09-13T11:53:06.266345+0000","last_fullsized":"2024-09-13T11:53:06.266345+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:05:51.189944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044149899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":21,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.271834+0000","last_change":"2024-09-13T11:53:06.272071+0000","last_active":"2024-09-13T11:53:06.271834+0000","last_peered":"2024-09-13T11:53:06.271834+0000","last_clean":"2024-09-13T11:53:06.271834+0000","last_became_active":"2024-09-13T11:53:04.270138+0000","last_became_peered":"2024-09-13T11:53:04.270138+0000","last_unstale":"2024-09-13T11:53:06.271834+0000","last_undegraded":"2024-09-13T11:53:06.271834+0000","last_fullsized":"2024-09-13T11:53:06.271834+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:17.090753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047405,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":111,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.265801+0000","last_change":"2024-09-13T11:52:57.706946+0000","last_active":"2024-09-13T11:53:06.265801+0000","last_peered":"2024-09-13T11:53:06.265801+0000","last_clean":"2024-09-13T11:53:06.265801+0000","last_became_active":"2024-09-13T11:52:57.705894+0000","last_became_peered":"2024-09-13T11:52:57.705894+0000","last_unstale":"2024-09-13T11:53:06.265801+0000","last_undegraded":"2024-09-13T11:53:06.265801+0000","last_fullsized":"2024-09-13T11:53:06.265801+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:52:56.225375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:52:56.225375+0000","last_clean_scrub_stamp":"2024-09-13T11:52:56.225375+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:05.299261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59246,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":172032,"data_stored":56599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607557,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":233472,"data_stored":585527,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607558,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":585527,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":176128,"data_stored":56618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":172032,"data_stored":56599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640262,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":176128,"data_stored":56618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:53:10.501 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T11:53:10.501 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T11:53:10.501 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-13T11:53:10.501 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T11:53:10.597 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.594+0000 7fae45423640 10 monclient: get_monmap_and_config 2024-09-13T11:53:10.598 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.594+0000 7fae45423640 10 monclient: build_initial_monmap 2024-09-13T11:53:10.598 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.594+0000 7fae45423640 10 monclient: monmap: 2024-09-13T11:53:10.598 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:10.598 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:10.598 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:10.596496+0000 2024-09-13T11:53:10.598 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:10.596496+0000 2024-09-13T11:53:10.598 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:10.598 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:10.598 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:10.598 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:10.598 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:10.599 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:10.599 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:10.599 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:10.599 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae45423640 1 Processor -- start 2024-09-13T11:53:10.599 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae45423640 1 -- start start 2024-09-13T11:53:10.599 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae45423640 10 monclient: init 2024-09-13T11:53:10.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae45423640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:10.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae45423640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-13T11:53:10.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae45423640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae4006ea50 0x7fae400a24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae45423640 10 monclient(hunting): picked mon.noname-b con 0x7fae4006ea50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:10.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae45423640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fae4006ee30 0x7fae4006f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae45423640 10 monclient(hunting): picked mon.noname-a con 0x7fae4006ee30 addr v2:172.21.15.100:3300/0 2024-09-13T11:53:10.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae45423640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fae40137710 0x7fae40137bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae45423640 10 monclient(hunting): picked mon.noname-c con 0x7fae40137710 addr v2:172.21.15.122:3300/0 2024-09-13T11:53:10.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae45423640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:10.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae45423640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7fae4006fd30 con 0x7fae4006ee30 2024-09-13T11:53:10.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae45423640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:10.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae45423640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7fae400a63e0 con 0x7fae40137710 2024-09-13T11:53:10.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae45423640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:10.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae45423640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fae400a4410 con 0x7fae4006ea50 2024-09-13T11:53:10.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae45423640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:10.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae45423640 10 monclient(hunting): authenticate will time out at 771.189822s 2024-09-13T11:53:10.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae3effd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae4006ea50 0x7fae400a24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae3e7fc640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fae4006ee30 0x7fae4006f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae3effd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae4006ea50 0x7fae400a24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47558/0 (socket says 172.21.15.100:47558) 2024-09-13T11:53:10.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae3e7fc640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fae4006ee30 0x7fae4006f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49714/0 (socket says 172.21.15.100:49714) 2024-09-13T11:53:10.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae3effd640 1 -- 172.21.15.100:0/1562839860 learned_addr learned my addr 172.21.15.100:0/1562839860 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:10.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae3effd640 10 monclient(hunting): get_auth_request con 0x7fae4006ea50 auth_method 0 2024-09-13T11:53:10.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:10.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae3effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:10.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:10.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae3e7fc640 10 monclient(hunting): get_auth_request con 0x7fae4006ee30 auth_method 0 2024-09-13T11:53:10.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae3e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:10.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae3e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:10.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.595+0000 7fae3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:10.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.596+0000 7fae3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:10.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.596+0000 7fae3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:10.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.596+0000 7fae3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:10.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.596+0000 7fae3e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:10.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.596+0000 7fae3e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:10.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.596+0000 7fae3e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:10.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.596+0000 7fae3f7fe640 1 --2- 172.21.15.100:0/1562839860 >> v2:172.21.15.122:3300/0 conn(0x7fae40137710 0x7fae40137bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.596+0000 7fae3f7fe640 10 monclient(hunting): get_auth_request con 0x7fae40137710 auth_method 0 2024-09-13T11:53:10.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.596+0000 7fae3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:10.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.596+0000 7fae3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:10.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.596+0000 7fae3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:10.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.596+0000 7fae3e7fc640 10 monclient(hunting): handle_auth_done global_id 4374 payload 274 2024-09-13T11:53:10.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.596+0000 7fae3e7fc640 1 -- 172.21.15.100:0/1562839860 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae4006ea50 msgr2=0x7fae400a24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:10.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.596+0000 7fae3e7fc640 1 --2- 172.21.15.100:0/1562839860 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae4006ea50 0x7fae400a24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.596+0000 7fae3e7fc640 1 -- 172.21.15.100:0/1562839860 >> v2:172.21.15.122:3300/0 conn(0x7fae40137710 msgr2=0x7fae40137bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:10.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.596+0000 7fae3e7fc640 1 --2- 172.21.15.100:0/1562839860 >> v2:172.21.15.122:3300/0 conn(0x7fae40137710 0x7fae40137bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.596+0000 7fae3e7fc640 10 monclient: _finish_hunting 0 2024-09-13T11:53:10.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.596+0000 7fae3e7fc640 1 monclient: found mon.noname-a 2024-09-13T11:53:10.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.596+0000 7fae3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:10.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.596+0000 7fae3e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:10.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.596+0000 7fae3e7fc640 1 -- 172.21.15.100:0/1562839860 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae400a65f0 con 0x7fae4006ee30 2024-09-13T11:53:10.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.596+0000 7fae3effd640 1 --2- 172.21.15.100:0/1562839860 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae4006ea50 0x7fae400a24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:10.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.596+0000 7fae3e7fc640 1 --2- 172.21.15.100:0/1562839860 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae4006ee30 0x7fae4006f210 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fae340097a0 tx=0x7fae3402a010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88da28e4b1ff1778 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:10.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae3dffb640 1 -- 172.21.15.100:0/1562839860 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae34037070 con 0x7fae4006ee30 2024-09-13T11:53:10.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:10.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae3dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:10.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae3dffb640 10 monclient: dump: 2024-09-13T11:53:10.614 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:10.614 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:10.614 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.615 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.615 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:10.615 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:10.615 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:10.615 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:10.615 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:10.615 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:10.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae3dffb640 10 monclient: _finish_auth 0 2024-09-13T11:53:10.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae3dffb640 10 monclient: _check_auth_tickets 2024-09-13T11:53:10.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:10.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae3dffb640 1 -- 172.21.15.100:0/1562839860 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae3402ad20 con 0x7fae4006ee30 2024-09-13T11:53:10.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae45423640 5 monclient: authenticate success, global_id 4374 2024-09-13T11:53:10.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae45423640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:10.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae3dffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:10.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae45423640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:10.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae3dffb640 1 -- 172.21.15.100:0/1562839860 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae34032660 con 0x7fae4006ee30 2024-09-13T11:53:10.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:10.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae45423640 4 set_mon_vals no callback set 2024-09-13T11:53:10.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae3dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:10.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae3dffb640 10 monclient: dump: 2024-09-13T11:53:10.616 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:10.616 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:10.616 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.616 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.616 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:10.616 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:10.617 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:10.617 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:10.617 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:10.617 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:10.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae45423640 10 monclient: shutdown 2024-09-13T11:53:10.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae45423640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:10.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae45423640 1 -- 172.21.15.100:0/1562839860 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae4006ee30 msgr2=0x7fae4006f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:10.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae45423640 1 --2- 172.21.15.100:0/1562839860 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae4006ee30 0x7fae4006f210 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fae340097a0 tx=0x7fae3402a010 comp rx=0 tx=0).stop 2024-09-13T11:53:10.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae45423640 1 -- 172.21.15.100:0/1562839860 shutdown_connections 2024-09-13T11:53:10.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae45423640 1 --2- 172.21.15.100:0/1562839860 >> v2:172.21.15.122:3300/0 conn(0x7fae40137710 0x7fae40137bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae45423640 1 --2- 172.21.15.100:0/1562839860 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae4006ee30 0x7fae4006f210 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae45423640 1 --2- 172.21.15.100:0/1562839860 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae4006ea50 0x7fae400a24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae45423640 1 -- 172.21.15.100:0/1562839860 >> 172.21.15.100:0/1562839860 conn(0x7fae400a6c30 msgr2=0x7fae400a7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:10.618 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae45423640 1 -- 172.21.15.100:0/1562839860 shutdown_connections 2024-09-13T11:53:10.618 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.597+0000 7fae45423640 1 -- 172.21.15.100:0/1562839860 wait complete. 2024-09-13T11:53:10.618 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.598+0000 7fae45423640 10 monclient: build_initial_monmap 2024-09-13T11:53:10.618 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.598+0000 7fae45423640 10 monclient: monmap: 2024-09-13T11:53:10.618 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:10.618 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:10.618 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:10.618 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:10.618 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:10.618 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:10.618 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:10.618 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:10.618 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:10.618 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:10.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.598+0000 7fae45423640 1 Processor -- start 2024-09-13T11:53:10.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.598+0000 7fae45423640 1 -- start start 2024-09-13T11:53:10.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.598+0000 7fae45423640 10 monclient: init 2024-09-13T11:53:10.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae45423640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:10.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae45423640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:53:10.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae45423640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae4006ea50 0x7fae401368a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae45423640 10 monclient(hunting): picked mon.noname-a con 0x7fae4006ea50 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:10.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae45423640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae4006ee30 0x7fae40136de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae45423640 10 monclient(hunting): picked mon.noname-b con 0x7fae4006ee30 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:10.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae45423640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae40137710 0x7fae401205a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae45423640 10 monclient(hunting): picked mon.noname-c con 0x7fae40137710 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:10.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae45423640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:10.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae45423640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fae4006f750 con 0x7fae4006ea50 2024-09-13T11:53:10.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae45423640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:10.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae45423640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fae400a5bd0 con 0x7fae4006ee30 2024-09-13T11:53:10.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae45423640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:10.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae45423640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fae401cd650 con 0x7fae40137710 2024-09-13T11:53:10.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae45423640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:10.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae45423640 10 monclient(hunting): authenticate will time out at 771.193230s 2024-09-13T11:53:10.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3effd640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae4006ea50 0x7fae401368a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3effd640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae4006ea50 0x7fae401368a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49728/0 (socket says 172.21.15.100:49728) 2024-09-13T11:53:10.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3effd640 1 -- 172.21.15.100:0/3624361107 learned_addr learned my addr 172.21.15.100:0/3624361107 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:10.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3effd640 10 monclient(hunting): get_auth_request con 0x7fae4006ea50 auth_method 0 2024-09-13T11:53:10.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:10.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:10.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:10.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3f7fe640 1 --2- 172.21.15.100:0/3624361107 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae40137710 0x7fae401205a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3f7fe640 10 monclient(hunting): get_auth_request con 0x7fae40137710 auth_method 0 2024-09-13T11:53:10.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:10.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:10.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:10.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:10.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:10.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:10.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:10.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:10.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:10.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3effd640 10 monclient(hunting): handle_auth_done global_id 4377 payload 824 2024-09-13T11:53:10.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3e7fc640 1 --2- 172.21.15.100:0/3624361107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae4006ee30 0x7fae40136de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3effd640 1 -- 172.21.15.100:0/3624361107 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae40137710 msgr2=0x7fae401205a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:10.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3effd640 1 --2- 172.21.15.100:0/3624361107 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae40137710 0x7fae401205a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3effd640 1 -- 172.21.15.100:0/3624361107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae4006ee30 msgr2=0x7fae40136de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:10.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3effd640 1 --2- 172.21.15.100:0/3624361107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae4006ee30 0x7fae40136de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3effd640 10 monclient: _finish_hunting 0 2024-09-13T11:53:10.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3effd640 1 monclient: found mon.noname-a 2024-09-13T11:53:10.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:10.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:10.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3effd640 1 -- 172.21.15.100:0/3624361107 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae40121f00 con 0x7fae4006ea50 2024-09-13T11:53:10.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3e7fc640 10 monclient: get_auth_request con 0x7fae4006ee30 auth_method 0 2024-09-13T11:53:10.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3e7fc640 1 --2- 172.21.15.100:0/3624361107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae4006ee30 0x7fae40136de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:53:10.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3effd640 1 --2- 172.21.15.100:0/3624361107 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae4006ea50 0x7fae401368a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fae3000e9d0 tx=0x7fae3000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:10.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae277fe640 1 -- 172.21.15.100:0/3624361107 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae30017070 con 0x7fae4006ea50 2024-09-13T11:53:10.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:10.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:10.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae277fe640 10 monclient: dump: 2024-09-13T11:53:10.624 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:10.624 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:10.624 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.624 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.624 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:10.624 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:10.624 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:10.624 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:10.624 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:10.625 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:10.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae277fe640 10 monclient: _finish_auth 0 2024-09-13T11:53:10.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae277fe640 10 monclient: _check_auth_tickets 2024-09-13T11:53:10.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:10.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae277fe640 1 -- 172.21.15.100:0/3624361107 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae3000ca60 con 0x7fae4006ea50 2024-09-13T11:53:10.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae45423640 5 monclient: authenticate success, global_id 4377 2024-09-13T11:53:10.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae277fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:10.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae277fe640 1 -- 172.21.15.100:0/3624361107 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae3000ce10 con 0x7fae4006ea50 2024-09-13T11:53:10.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae45423640 10 monclient: _renew_subs 2024-09-13T11:53:10.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae45423640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:10.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae45423640 1 -- 172.21.15.100:0/3624361107 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae400a3480 con 0x7fae4006ea50 2024-09-13T11:53:10.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:10.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:10.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae277fe640 10 monclient: dump: 2024-09-13T11:53:10.626 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:10.626 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:10.626 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.626 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.626 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:10.626 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:10.626 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:10.626 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:10.626 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:10.626 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:10.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae3d7fa640 4 set_mon_vals no callback set 2024-09-13T11:53:10.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae45423640 10 monclient: _renew_subs 2024-09-13T11:53:10.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae45423640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:10.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.599+0000 7fae45423640 1 -- 172.21.15.100:0/3624361107 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae401239c0 con 0x7fae4006ea50 2024-09-13T11:53:10.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.600+0000 7fae277fe640 1 -- 172.21.15.100:0/3624361107 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fae3001a070 con 0x7fae4006ea50 2024-09-13T11:53:10.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.600+0000 7fae45423640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:10.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.600+0000 7fae45423640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:10.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.600+0000 7fae45423640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:10.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.600+0000 7fae45423640 1 -- 172.21.15.100:0/3624361107 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae401382c0 con 0x7fae4006ea50 2024-09-13T11:53:10.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.600+0000 7fae277fe640 1 --2- 172.21.15.100:0/3624361107 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fae2c03dfa0 0x7fae2c040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.601+0000 7fae277fe640 1 -- 172.21.15.100:0/3624361107 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4402+0+0 (secure 0 0 0) 0x7fae30052560 con 0x7fae4006ea50 2024-09-13T11:53:10.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.601+0000 7fae3e7fc640 1 --2- 172.21.15.100:0/3624361107 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fae2c03dfa0 0x7fae2c040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.601+0000 7fae3e7fc640 10 monclient: get_auth_request con 0x7fae2c03dfa0 auth_method 0 2024-09-13T11:53:10.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.601+0000 7fae3e7fc640 1 --2- 172.21.15.100:0/3624361107 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fae2c03dfa0 0x7fae2c040460 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fae34034000 tx=0x7fae3402e000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:10.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.605+0000 7fae277fe640 1 -- 172.21.15.100:0/3624361107 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fae30013c70 con 0x7fae4006ea50 2024-09-13T11:53:10.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.605+0000 7fae277fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:10.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.605+0000 7fae277fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:10.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.814+0000 7fae45423640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-13T11:53:10.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.814+0000 7fae45423640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-13T11:53:10.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.814+0000 7fae45423640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:10.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.814+0000 7fae45423640 1 -- 172.21.15.100:0/3624361107 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fae400a10c0 con 0x7fae4006ea50 2024-09-13T11:53:10.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.814+0000 7fae277fe640 1 -- 172.21.15.100:0/3624361107 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fae30052230 con 0x7fae4006ea50 2024-09-13T11:53:10.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.814+0000 7fae277fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-13T11:53:10.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.814+0000 7fae277fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:10.817 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:53:10.817 INFO:teuthology.orchestra.run.smithi100.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T11:53:10.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.816+0000 7fae45423640 1 -- 172.21.15.100:0/3624361107 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fae2c03dfa0 msgr2=0x7fae2c040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:10.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.816+0000 7fae45423640 1 --2- 172.21.15.100:0/3624361107 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fae2c03dfa0 0x7fae2c040460 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fae34034000 tx=0x7fae3402e000 comp rx=0 tx=0).stop 2024-09-13T11:53:10.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.816+0000 7fae45423640 10 monclient: shutdown 2024-09-13T11:53:10.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.816+0000 7fae45423640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:10.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.816+0000 7fae45423640 1 -- 172.21.15.100:0/3624361107 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae4006ea50 msgr2=0x7fae401368a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:10.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.816+0000 7fae45423640 1 --2- 172.21.15.100:0/3624361107 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae4006ea50 0x7fae401368a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fae3000e9d0 tx=0x7fae3000ee90 comp rx=0 tx=0).stop 2024-09-13T11:53:10.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.816+0000 7fae45423640 1 -- 172.21.15.100:0/3624361107 shutdown_connections 2024-09-13T11:53:10.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.816+0000 7fae45423640 1 --2- 172.21.15.100:0/3624361107 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fae2c03dfa0 0x7fae2c040460 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.816+0000 7fae45423640 1 --2- 172.21.15.100:0/3624361107 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae40137710 0x7fae401205a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.816+0000 7fae45423640 1 --2- 172.21.15.100:0/3624361107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae4006ee30 0x7fae40136de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.816+0000 7fae45423640 1 --2- 172.21.15.100:0/3624361107 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae4006ea50 0x7fae401368a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.816+0000 7fae45423640 1 -- 172.21.15.100:0/3624361107 >> 172.21.15.100:0/3624361107 conn(0x7fae400a6c30 msgr2=0x7fae401bf1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:10.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.816+0000 7fae45423640 1 -- 172.21.15.100:0/3624361107 shutdown_connections 2024-09-13T11:53:10.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.816+0000 7fae45423640 1 -- 172.21.15.100:0/3624361107 wait complete. 2024-09-13T11:53:10.829 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-13T11:53:10.829 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-13T11:53:10.844 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-13T11:53:10.845 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T11:53:10.943 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.940+0000 7fd535390640 10 monclient: get_monmap_and_config 2024-09-13T11:53:10.943 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.940+0000 7fd535390640 10 monclient: build_initial_monmap 2024-09-13T11:53:10.943 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.940+0000 7fd535390640 10 monclient: monmap: 2024-09-13T11:53:10.943 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:10.943 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:10.944 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:10.942505+0000 2024-09-13T11:53:10.944 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:10.942505+0000 2024-09-13T11:53:10.944 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:10.944 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:10.944 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:10.944 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:10.944 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:10.945 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:10.945 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:10.945 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:10.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.941+0000 7fd535390640 1 Processor -- start 2024-09-13T11:53:10.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.941+0000 7fd535390640 1 -- start start 2024-09-13T11:53:10.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.941+0000 7fd535390640 10 monclient: init 2024-09-13T11:53:10.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.941+0000 7fd535390640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:10.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.941+0000 7fd535390640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-13T11:53:10.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.941+0000 7fd535390640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd53006ea50 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:10.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.941+0000 7fd535390640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd5300a2860 0x7fd53006f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.941+0000 7fd535390640 10 monclient(hunting): picked mon.noname-a con 0x7fd5300a2860 addr v2:172.21.15.100:3300/0 2024-09-13T11:53:10.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.941+0000 7fd535390640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd53013c9f0 0x7fd53013ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.941+0000 7fd535390640 10 monclient(hunting): picked mon.noname-b con 0x7fd53013c9f0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:10.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.941+0000 7fd535390640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd53006fa80 con 0x7fd53006ea50 2024-09-13T11:53:10.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.941+0000 7fd535390640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:10.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.941+0000 7fd535390640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7fd5300a2cb0 con 0x7fd5300a2860 2024-09-13T11:53:10.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.941+0000 7fd535390640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:10.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.941+0000 7fd535390640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fd5300a15e0 con 0x7fd53013c9f0 2024-09-13T11:53:10.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.941+0000 7fd535390640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:10.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.941+0000 7fd535390640 10 monclient(hunting): authenticate will time out at 771.535931s 2024-09-13T11:53:10.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd527fff640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd5300a2860 0x7fd53006f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd52f7fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd53013c9f0 0x7fd53013ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd527fff640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd5300a2860 0x7fd53006f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49738/0 (socket says 172.21.15.100:49738) 2024-09-13T11:53:10.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd527fff640 1 -- 172.21.15.100:0/3003496522 learned_addr learned my addr 172.21.15.100:0/3003496522 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:10.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd527fff640 10 monclient(hunting): get_auth_request con 0x7fd5300a2860 auth_method 0 2024-09-13T11:53:10.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd527fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:10.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd527fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:10.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd527fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:10.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd52f7fe640 10 monclient(hunting): get_auth_request con 0x7fd53013c9f0 auth_method 0 2024-09-13T11:53:10.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd52f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:10.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd52f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:10.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd52f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:10.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd527fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:10.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd527fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:10.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd527fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:10.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd52f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:10.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd52f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:10.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd52f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:10.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd527fff640 10 monclient(hunting): handle_auth_done global_id 4380 payload 274 2024-09-13T11:53:10.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd527fff640 1 -- 172.21.15.100:0/3003496522 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd53013c9f0 msgr2=0x7fd53013ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:10.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd527fff640 1 --2- 172.21.15.100:0/3003496522 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd53013c9f0 0x7fd53013ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd527fff640 1 -- 172.21.15.100:0/3003496522 >> v1:172.21.15.122:6789/0 conn(0x7fd53006ea50 legacy=0x7fd53006ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:10.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd527fff640 10 monclient: _finish_hunting 0 2024-09-13T11:53:10.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd527fff640 1 monclient: found mon.noname-a 2024-09-13T11:53:10.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd527fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:10.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd527fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:10.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd527fff640 1 -- 172.21.15.100:0/3003496522 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5300a1870 con 0x7fd5300a2860 2024-09-13T11:53:10.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd52f7fe640 1 --2- 172.21.15.100:0/3003496522 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd53013c9f0 0x7fd53013ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:10.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.942+0000 7fd527fff640 1 --2- 172.21.15.100:0/3003496522 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5300a2860 0x7fd53006f540 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd5180098a0 tx=0x7fd51802a160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=185c9a89edd9a488 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:10.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd52e7fc640 1 -- 172.21.15.100:0/3003496522 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd518037070 con 0x7fd5300a2860 2024-09-13T11:53:10.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd52e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:10.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd52e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:10.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd52e7fc640 10 monclient: dump: 2024-09-13T11:53:10.967 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:10.967 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:10.968 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.968 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.968 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:10.968 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:10.968 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:10.969 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:10.969 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:10.969 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:10.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd52e7fc640 10 monclient: _finish_auth 0 2024-09-13T11:53:10.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd52e7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:53:10.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd52e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:10.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd52e7fc640 1 -- 172.21.15.100:0/3003496522 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd518032440 con 0x7fd5300a2860 2024-09-13T11:53:10.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd535390640 5 monclient: authenticate success, global_id 4380 2024-09-13T11:53:10.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd535390640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:10.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd52e7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:10.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd535390640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:10.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd52e7fc640 1 -- 172.21.15.100:0/3003496522 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd518032810 con 0x7fd5300a2860 2024-09-13T11:53:10.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd52e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:10.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd535390640 4 set_mon_vals no callback set 2024-09-13T11:53:10.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd52e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:10.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd52e7fc640 10 monclient: dump: 2024-09-13T11:53:10.971 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:10.972 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:10.972 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.972 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.972 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:10.972 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:10.973 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:10.973 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:10.973 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:10.973 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:10.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd535390640 10 monclient: shutdown 2024-09-13T11:53:10.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd535390640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:10.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd535390640 1 -- 172.21.15.100:0/3003496522 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5300a2860 msgr2=0x7fd53006f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:10.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd535390640 1 --2- 172.21.15.100:0/3003496522 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5300a2860 0x7fd53006f540 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd5180098a0 tx=0x7fd51802a160 comp rx=0 tx=0).stop 2024-09-13T11:53:10.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd535390640 1 -- 172.21.15.100:0/3003496522 shutdown_connections 2024-09-13T11:53:10.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd535390640 1 --2- 172.21.15.100:0/3003496522 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd53013c9f0 0x7fd53013ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd535390640 1 --2- 172.21.15.100:0/3003496522 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5300a2860 0x7fd53006f540 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.975 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd535390640 1 -- 172.21.15.100:0/3003496522 >> 172.21.15.100:0/3003496522 conn(0x7fd5300a68e0 msgr2=0x7fd5300a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:10.975 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd535390640 1 -- 172.21.15.100:0/3003496522 shutdown_connections 2024-09-13T11:53:10.975 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.943+0000 7fd535390640 1 -- 172.21.15.100:0/3003496522 wait complete. 2024-09-13T11:53:10.975 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.944+0000 7fd535390640 10 monclient: build_initial_monmap 2024-09-13T11:53:10.975 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.944+0000 7fd535390640 10 monclient: monmap: 2024-09-13T11:53:10.976 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:10.976 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:10.976 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:10.976 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:10.976 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:10.976 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:10.977 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:10.977 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:10.977 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:10.977 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:10.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.944+0000 7fd535390640 1 Processor -- start 2024-09-13T11:53:10.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.944+0000 7fd535390640 1 -- start start 2024-09-13T11:53:10.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.944+0000 7fd535390640 10 monclient: init 2024-09-13T11:53:10.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd535390640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:10.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd535390640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:53:10.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd535390640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd53006ea50 0x7fd5301cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd535390640 10 monclient(hunting): picked mon.noname-c con 0x7fd53006ea50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:10.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd535390640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5300a2860 0x7fd530137900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd535390640 10 monclient(hunting): picked mon.noname-b con 0x7fd5300a2860 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:10.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd535390640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd53013c9f0 0x7fd53009ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd535390640 10 monclient(hunting): picked mon.noname-a con 0x7fd53013c9f0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:10.980 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd535390640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:10.980 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd535390640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fd5300a5be0 con 0x7fd53013c9f0 2024-09-13T11:53:10.980 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd535390640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:10.980 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd535390640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fd5300a5a60 con 0x7fd5300a2860 2024-09-13T11:53:10.980 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd535390640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:10.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd535390640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fd53013f320 con 0x7fd53006ea50 2024-09-13T11:53:10.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd535390640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:10.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd535390640 10 monclient(hunting): authenticate will time out at 771.539223s 2024-09-13T11:53:10.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52effd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd53006ea50 0x7fd5301cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52f7fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd53013c9f0 0x7fd53009ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52effd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd53006ea50 0x7fd5301cc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47572/0 (socket says 172.21.15.100:47572) 2024-09-13T11:53:10.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52effd640 1 -- 172.21.15.100:0/3616018378 learned_addr learned my addr 172.21.15.100:0/3616018378 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:10.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52effd640 10 monclient(hunting): get_auth_request con 0x7fd53006ea50 auth_method 0 2024-09-13T11:53:10.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:10.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:10.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:10.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52f7fe640 10 monclient(hunting): get_auth_request con 0x7fd53013c9f0 auth_method 0 2024-09-13T11:53:10.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:10.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:10.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:10.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:10.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:10.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd527fff640 1 --2- 172.21.15.100:0/3616018378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5300a2860 0x7fd530137900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:10.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:10.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:10.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:10.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd527fff640 10 monclient(hunting): get_auth_request con 0x7fd5300a2860 auth_method 0 2024-09-13T11:53:10.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd527fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:10.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd527fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:10.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd527fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:10.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd527fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:10.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd527fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:10.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd527fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:10.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52f7fe640 10 monclient(hunting): handle_auth_done global_id 4383 payload 824 2024-09-13T11:53:10.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52f7fe640 1 -- 172.21.15.100:0/3616018378 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd53006ea50 msgr2=0x7fd5301cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:10.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52f7fe640 1 --2- 172.21.15.100:0/3616018378 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd53006ea50 0x7fd5301cc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52f7fe640 1 -- 172.21.15.100:0/3616018378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5300a2860 msgr2=0x7fd530137900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:10.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52f7fe640 1 --2- 172.21.15.100:0/3616018378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5300a2860 0x7fd530137900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:10.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52f7fe640 10 monclient: _finish_hunting 0 2024-09-13T11:53:10.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52f7fe640 1 monclient: found mon.noname-a 2024-09-13T11:53:10.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:10.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:10.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52f7fe640 1 -- 172.21.15.100:0/3616018378 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5301cd4f0 con 0x7fd53013c9f0 2024-09-13T11:53:10.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52effd640 1 --2- 172.21.15.100:0/3616018378 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd53006ea50 0x7fd5301cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:10.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52f7fe640 1 --2- 172.21.15.100:0/3616018378 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd53013c9f0 0x7fd53009ece0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd52800efc0 tx=0x7fd52800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:10.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd5277fe640 1 -- 172.21.15.100:0/3616018378 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd528018030 con 0x7fd53013c9f0 2024-09-13T11:53:10.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd5277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:10.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd5277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:10.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd5277fe640 10 monclient: dump: 2024-09-13T11:53:10.989 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:10.989 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:10.990 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.990 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.990 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:10.990 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:10.990 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:10.990 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:10.991 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:10.991 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:10.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd5277fe640 10 monclient: _finish_auth 0 2024-09-13T11:53:10.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd5277fe640 10 monclient: _check_auth_tickets 2024-09-13T11:53:10.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd5277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:10.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd5277fe640 1 -- 172.21.15.100:0/3616018378 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd52800cd30 con 0x7fd53013c9f0 2024-09-13T11:53:10.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd535390640 5 monclient: authenticate success, global_id 4383 2024-09-13T11:53:10.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd5277fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:10.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd535390640 10 monclient: _renew_subs 2024-09-13T11:53:10.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd5277fe640 1 -- 172.21.15.100:0/3616018378 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd528003bb0 con 0x7fd53013c9f0 2024-09-13T11:53:10.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd535390640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:10.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd535390640 1 -- 172.21.15.100:0/3616018378 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd530122290 con 0x7fd53013c9f0 2024-09-13T11:53:10.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd5277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:10.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd5277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:10.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd5277fe640 10 monclient: dump: 2024-09-13T11:53:10.994 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:10.994 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:10.994 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.994 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:10.994 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:10.995 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:10.995 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:10.995 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:10.995 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:10.995 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:10.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd52dffb640 4 set_mon_vals no callback set 2024-09-13T11:53:10.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd535390640 10 monclient: _renew_subs 2024-09-13T11:53:10.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd535390640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:10.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.945+0000 7fd535390640 1 -- 172.21.15.100:0/3616018378 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5300a0760 con 0x7fd53013c9f0 2024-09-13T11:53:10.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.946+0000 7fd5277fe640 1 -- 172.21.15.100:0/3616018378 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fd528005d10 con 0x7fd53013c9f0 2024-09-13T11:53:10.997 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.946+0000 7fd535390640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:10.997 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.946+0000 7fd535390640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:10.997 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.946+0000 7fd535390640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:10.997 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.946+0000 7fd535390640 1 -- 172.21.15.100:0/3616018378 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd53006ee30 con 0x7fd53013c9f0 2024-09-13T11:53:10.997 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.946+0000 7fd5277fe640 1 --2- 172.21.15.100:0/3616018378 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fd51003dfa0 0x7fd510040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:10.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.947+0000 7fd5277fe640 1 -- 172.21.15.100:0/3616018378 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4374+0+0 (secure 0 0 0) 0x7fd528053000 con 0x7fd53013c9f0 2024-09-13T11:53:10.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.947+0000 7fd52effd640 1 --2- 172.21.15.100:0/3616018378 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fd51003dfa0 0x7fd510040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:10.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.947+0000 7fd52effd640 10 monclient: get_auth_request con 0x7fd51003dfa0 auth_method 0 2024-09-13T11:53:10.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.947+0000 7fd52effd640 1 --2- 172.21.15.100:0/3616018378 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fd51003dfa0 0x7fd510040460 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd520000f60 tx=0x7fd520009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:10.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.951+0000 7fd5277fe640 1 -- 172.21.15.100:0/3616018378 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fd528022070 con 0x7fd53013c9f0 2024-09-13T11:53:10.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.951+0000 7fd5277fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:10.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:10.951+0000 7fd5277fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:11.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.125+0000 7fd535390640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:53:11.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.125+0000 7fd535390640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:53:11.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.125+0000 7fd535390640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:11.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.125+0000 7fd535390640 1 -- 172.21.15.100:0/3616018378 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd530069100 con 0x7fd53013c9f0 2024-09-13T11:53:11.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.125+0000 7fd5277fe640 1 -- 172.21.15.100:0/3616018378 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+12358 (secure 0 0 0) 0x7fd528010040 con 0x7fd53013c9f0 2024-09-13T11:53:11.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.126+0000 7fd5277fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:53:11.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.126+0000 7fd5277fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:11.128 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:53:11.128 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":19,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","created":"2024-09-13T11:52:46.158794+0000","modified":"2024-09-13T11:53:10.936904+0000","last_up_change":"2024-09-13T11:52:56.225375+0000","last_in_change":"2024-09-13T11:52:47.800207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T11:52:55.233767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T11:53:03.003491+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac784459-dbff-42ed-941e-70c741392dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":92840425},{"type":"v1","addr":"172.21.15.100:6817","nonce":92840425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":92840425},{"type":"v1","addr":"172.21.15.100:6819","nonce":92840425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6822","nonce":92840425},{"type":"v1","addr":"172.21.15.100:6823","nonce":92840425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":92840425},{"type":"v1","addr":"172.21.15.100:6821","nonce":92840425}]},"public_addr":"172.21.15.100:6817/92840425","cluster_addr":"172.21.15.100:6819/92840425","heartbeat_back_addr":"172.21.15.100:6823/92840425","heartbeat_front_addr":"172.21.15.100:6821/92840425","state":["exists","up"]},{"osd":1,"uuid":"8d8d39e7-e8f1-499a-a01b-e7ba49844eae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":1385319817},{"type":"v1","addr":"172.21.15.100:6809","nonce":1385319817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":1385319817},{"type":"v1","addr":"172.21.15.100:6811","nonce":1385319817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":1385319817},{"type":"v1","addr":"172.21.15.100:6815","nonce":1385319817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":1385319817},{"type":"v1","addr":"172.21.15.100:6813","nonce":1385319817}]},"public_addr":"172.21.15.100:6809/1385319817","cluster_addr":"172.21.15.100:6811/1385319817","heartbeat_back_addr":"172.21.15.100:6815/1385319817","heartbeat_front_addr":"172.21.15.100:6813/1385319817","state":["exists","up"]},{"osd":2,"uuid":"e0bd3e30-0ff3-4067-83d3-ef8d84b29097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":2354314235},{"type":"v1","addr":"172.21.15.100:6801","nonce":2354314235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":2354314235},{"type":"v1","addr":"172.21.15.100:6803","nonce":2354314235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":2354314235},{"type":"v1","addr":"172.21.15.100:6807","nonce":2354314235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":2354314235},{"type":"v1","addr":"172.21.15.100:6805","nonce":2354314235}]},"public_addr":"172.21.15.100:6801/2354314235","cluster_addr":"172.21.15.100:6803/2354314235","heartbeat_back_addr":"172.21.15.100:6807/2354314235","heartbeat_front_addr":"172.21.15.100:6805/2354314235","state":["exists","up"]},{"osd":3,"uuid":"eca4429b-38cd-488a-b31d-c5856d701bde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3206511700},{"type":"v1","addr":"172.21.15.122:6809","nonce":3206511700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3206511700},{"type":"v1","addr":"172.21.15.122:6811","nonce":3206511700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3206511700},{"type":"v1","addr":"172.21.15.122:6815","nonce":3206511700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3206511700},{"type":"v1","addr":"172.21.15.122:6813","nonce":3206511700}]},"public_addr":"172.21.15.122:6809/3206511700","cluster_addr":"172.21.15.122:6811/3206511700","heartbeat_back_addr":"172.21.15.122:6815/3206511700","heartbeat_front_addr":"172.21.15.122:6813/3206511700","state":["exists","up"]},{"osd":4,"uuid":"b7ddc8f1-38d5-496a-9bf7-181eab7e5516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":777240702},{"type":"v1","addr":"172.21.15.122:6817","nonce":777240702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":777240702},{"type":"v1","addr":"172.21.15.122:6819","nonce":777240702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":777240702},{"type":"v1","addr":"172.21.15.122:6823","nonce":777240702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":777240702},{"type":"v1","addr":"172.21.15.122:6821","nonce":777240702}]},"public_addr":"172.21.15.122:6817/777240702","cluster_addr":"172.21.15.122:6819/777240702","heartbeat_back_addr":"172.21.15.122:6823/777240702","heartbeat_front_addr":"172.21.15.122:6821/777240702","state":["exists","up"]},{"osd":5,"uuid":"9010f8e0-c97b-451c-8b29-3ccb99e982a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":904725860},{"type":"v1","addr":"172.21.15.122:6801","nonce":904725860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":904725860},{"type":"v1","addr":"172.21.15.122:6803","nonce":904725860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":904725860},{"type":"v1","addr":"172.21.15.122:6807","nonce":904725860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":904725860},{"type":"v1","addr":"172.21.15.122:6805","nonce":904725860}]},"public_addr":"172.21.15.122:6801/904725860","cluster_addr":"172.21.15.122:6803/904725860","heartbeat_back_addr":"172.21.15.122:6807/904725860","heartbeat_front_addr":"172.21.15.122:6805/904725860","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:54.752725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:53.791663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:53.535397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:54.348980+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:54.658725+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:53.932256+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:53:11.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.128+0000 7fd535390640 1 -- 172.21.15.100:0/3616018378 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fd51003dfa0 msgr2=0x7fd510040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:11.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.128+0000 7fd535390640 1 --2- 172.21.15.100:0/3616018378 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fd51003dfa0 0x7fd510040460 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd520000f60 tx=0x7fd520009040 comp rx=0 tx=0).stop 2024-09-13T11:53:11.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.128+0000 7fd535390640 10 monclient: shutdown 2024-09-13T11:53:11.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.128+0000 7fd535390640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:11.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.128+0000 7fd535390640 1 -- 172.21.15.100:0/3616018378 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd53013c9f0 msgr2=0x7fd53009ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:11.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.128+0000 7fd535390640 1 --2- 172.21.15.100:0/3616018378 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd53013c9f0 0x7fd53009ece0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd52800efc0 tx=0x7fd52800c490 comp rx=0 tx=0).stop 2024-09-13T11:53:11.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.128+0000 7fd535390640 1 -- 172.21.15.100:0/3616018378 shutdown_connections 2024-09-13T11:53:11.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.128+0000 7fd535390640 1 --2- 172.21.15.100:0/3616018378 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fd51003dfa0 0x7fd510040460 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.128+0000 7fd535390640 1 --2- 172.21.15.100:0/3616018378 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd53013c9f0 0x7fd53009ece0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.128+0000 7fd535390640 1 --2- 172.21.15.100:0/3616018378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5300a2860 0x7fd530137900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.128+0000 7fd535390640 1 --2- 172.21.15.100:0/3616018378 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd53006ea50 0x7fd5301cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.128+0000 7fd535390640 1 -- 172.21.15.100:0/3616018378 >> 172.21.15.100:0/3616018378 conn(0x7fd5300a68e0 msgr2=0x7fd53013d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:11.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.128+0000 7fd535390640 1 -- 172.21.15.100:0/3616018378 shutdown_connections 2024-09-13T11:53:11.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.128+0000 7fd535390640 1 -- 172.21.15.100:0/3616018378 wait complete. 2024-09-13T11:53:11.141 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T11:52:55.233767+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-13T11:53:03.003491+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 18, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T11:53:11.142 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T11:53:11.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.240+0000 7f85d1575640 10 monclient: get_monmap_and_config 2024-09-13T11:53:11.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.240+0000 7f85d1575640 10 monclient: build_initial_monmap 2024-09-13T11:53:11.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.240+0000 7f85d1575640 10 monclient: monmap: 2024-09-13T11:53:11.316 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:11.316 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:11.316 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:11.241881+0000 2024-09-13T11:53:11.316 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:11.241881+0000 2024-09-13T11:53:11.317 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:11.317 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:11.317 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:11.317 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:11.317 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:11.317 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:11.317 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:11.317 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:11.328 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.240+0000 7f85d1575640 1 Processor -- start 2024-09-13T11:53:11.329 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.240+0000 7f85d1575640 1 -- start start 2024-09-13T11:53:11.329 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.240+0000 7f85d1575640 10 monclient: init 2024-09-13T11:53:11.329 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85d1575640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:11.329 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85d1575640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-13T11:53:11.330 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85d1575640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f85cc0a32e0 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:11.330 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85d1575640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f85cc0a2ae0 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:11.330 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85d1575640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f85cc13e070 0x7f85cc140460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:11.330 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85d1575640 10 monclient(hunting): picked mon.noname-c con 0x7f85cc13e070 addr v2:172.21.15.122:3300/0 2024-09-13T11:53:11.330 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85d1575640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85cc0708c0 con 0x7f85cc0a2ae0 2024-09-13T11:53:11.331 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85d1575640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85cc0a1660 con 0x7f85cc0a32e0 2024-09-13T11:53:11.331 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85d1575640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:11.331 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85d1575640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f85cc0a2ef0 con 0x7f85cc13e070 2024-09-13T11:53:11.331 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85d1575640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:11.331 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85d1575640 10 monclient(hunting): authenticate will time out at 771.835220s 2024-09-13T11:53:11.331 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85c27fc640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f85cc0a2ae0 0x7f85cc13c950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:58038/0 (socket says 172.21.15.100:58038) 2024-09-13T11:53:11.331 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85c27fc640 1 -- 172.21.15.100:0/406195307 learned_addr learned my addr 172.21.15.100:0/406195307 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:53:11.331 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 1 -- 172.21.15.100:0/406195307 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 789834272 0 0) 0x7f85cc0708c0 con 0x7f85cc0a2ae0 2024-09-13T11:53:11.331 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85cb7fe640 1 --2- 172.21.15.100:0/406195307 >> v2:172.21.15.122:3300/0 conn(0x7f85cc13e070 0x7f85cc140460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:11.331 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 10 monclient(hunting): _negotiate 2024-09-13T11:53:11.331 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:11.331 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:11.331 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 10 monclient(hunting): my global_id is 4386 2024-09-13T11:53:11.332 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 1 -- 172.21.15.100:0/406195307 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85b8003520 con 0x7f85cc0a2ae0 2024-09-13T11:53:11.332 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85cb7fe640 10 monclient(hunting): get_auth_request con 0x7f85cc13e070 auth_method 0 2024-09-13T11:53:11.332 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:11.332 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:11.332 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:11.332 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 1 -- 172.21.15.100:0/406195307 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2536673800 0 0) 0x7f85b8003520 con 0x7f85cc0a2ae0 2024-09-13T11:53:11.332 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 1 -- 172.21.15.100:0/406195307 >> v2:172.21.15.122:3300/0 conn(0x7f85cc13e070 msgr2=0x7f85cc140460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:11.332 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 1 --2- 172.21.15.100:0/406195307 >> v2:172.21.15.122:3300/0 conn(0x7f85cc13e070 0x7f85cc140460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.332 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 1 -- 172.21.15.100:0/406195307 >> v1:172.21.15.122:6789/0 conn(0x7f85cc0a32e0 legacy=0x7f85cc0a36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:11.332 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 10 monclient: _finish_hunting 0 2024-09-13T11:53:11.332 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 1 monclient: found mon.noname-a-legacy 2024-09-13T11:53:11.332 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:11.332 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.100:6789/0 2024-09-13T11:53:11.333 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 1 -- 172.21.15.100:0/406195307 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85cc0a1940 con 0x7f85cc0a2ae0 2024-09-13T11:53:11.333 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 10 monclient: _finish_auth 0 2024-09-13T11:53:11.333 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:53:11.333 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:11.333 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85d1575640 5 monclient: authenticate success, global_id 4386 2024-09-13T11:53:11.333 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 1 -- 172.21.15.100:0/406195307 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 626700363 0 0) 0x7f85b40016f0 con 0x7f85cc0a2ae0 2024-09-13T11:53:11.333 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85d1575640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:11.333 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85cb7fe640 1 --2- 172.21.15.100:0/406195307 >> v2:172.21.15.122:3300/0 conn(0x7f85cc13e070 0x7f85cc140460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:53:11.333 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:11.333 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-13T11:53:11.333 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 10 monclient: dump: 2024-09-13T11:53:11.334 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:11.334 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:11.334 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.334 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.334 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:11.334 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:11.334 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:11.334 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:11.334 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:11.334 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:11.334 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] but i'm connected to v1:172.21.15.100:6789/0, reconnecting 2024-09-13T11:53:11.334 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:11.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 1 -- 172.21.15.100:0/406195307 >> v1:172.21.15.100:6789/0 conn(0x7f85cc0a2ae0 legacy=0x7f85cc13c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:11.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:53:11.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 1 --2- 172.21.15.100:0/406195307 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85b8003a50 0x7f85b8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:11.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 10 monclient(hunting): picked mon.c con 0x7f85b8003a50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:11.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 1 --2- 172.21.15.100:0/406195307 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85b8004800 0x7f85b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:11.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 10 monclient(hunting): picked mon.b con 0x7f85b8004800 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:11.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 1 --2- 172.21.15.100:0/406195307 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85b80053a0 0x7f85b800ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:11.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 10 monclient(hunting): picked mon.a con 0x7f85b80053a0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:11.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:11.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 1 -- 172.21.15.100:0/406195307 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc0a2ef0 con 0x7f85b80053a0 2024-09-13T11:53:11.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:11.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 1 -- 172.21.15.100:0/406195307 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f85b800c2e0 con 0x7f85b8004800 2024-09-13T11:53:11.335 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:11.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 1 -- 172.21.15.100:0/406195307 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f85b800c4e0 con 0x7f85b8003a50 2024-09-13T11:53:11.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:11.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85ca7fc640 10 monclient(hunting): _finish_auth 0 2024-09-13T11:53:11.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.241+0000 7f85d1575640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:11.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85caffd640 1 --2- 172.21.15.100:0/406195307 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85b8003a50 0x7f85b8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:11.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85cb7fe640 1 --2- 172.21.15.100:0/406195307 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85b80053a0 0x7f85b800ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:11.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85caffd640 10 monclient(hunting): get_auth_request con 0x7f85b8003a50 auth_method 0 2024-09-13T11:53:11.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:11.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85caffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:11.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85caffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:53:11.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85c27fc640 1 --2- 172.21.15.100:0/406195307 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85b8004800 0x7f85b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:11.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85cb7fe640 10 monclient(hunting): get_auth_request con 0x7f85b80053a0 auth_method 0 2024-09-13T11:53:11.336 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:11.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:11.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85cb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:53:11.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85c27fc640 10 monclient(hunting): get_auth_request con 0x7f85b8004800 auth_method 0 2024-09-13T11:53:11.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85c27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:11.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85c27fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:11.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85c27fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:53:11.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:11.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:11.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85caffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:53:11.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:11.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:11.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:53:11.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85c27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:11.338 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85c27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:11.338 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85c27fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:53:11.338 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85caffd640 10 monclient(hunting): handle_auth_done global_id 4386 payload 293 2024-09-13T11:53:11.338 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85caffd640 1 -- 172.21.15.100:0/406195307 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85b8004800 msgr2=0x7f85b8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:11.338 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85caffd640 1 --2- 172.21.15.100:0/406195307 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85b8004800 0x7f85b8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.338 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85caffd640 1 -- 172.21.15.100:0/406195307 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85b80053a0 msgr2=0x7f85b800ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:11.338 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85caffd640 1 --2- 172.21.15.100:0/406195307 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85b80053a0 0x7f85b800ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.338 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85caffd640 10 monclient: _finish_hunting 0 2024-09-13T11:53:11.338 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85caffd640 1 monclient: found mon.c 2024-09-13T11:53:11.338 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:11.338 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85caffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:11.338 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85caffd640 1 -- 172.21.15.100:0/406195307 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85b8003520 con 0x7f85b8003a50 2024-09-13T11:53:11.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85caffd640 10 monclient: _finish_auth 0 2024-09-13T11:53:11.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85caffd640 10 monclient: _check_auth_tickets 2024-09-13T11:53:11.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:11.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85cb7fe640 1 --2- 172.21.15.100:0/406195307 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85b80053a0 0x7f85b800ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:11.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85cb7fe640 1 -- 172.21.15.100:0/406195307 reap_dead start 2024-09-13T11:53:11.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.242+0000 7f85caffd640 1 --2- 172.21.15.100:0/406195307 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85b8003a50 0x7f85b8003e70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f85bc001790 tx=0x7f85b8005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=995f8e69bb81d182 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:11.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.243+0000 7f85ca7fc640 1 -- 172.21.15.100:0/406195307 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85bc0013b0 con 0x7f85b8003a50 2024-09-13T11:53:11.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.243+0000 7f85ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:11.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.243+0000 7f85ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:11.339 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.243+0000 7f85ca7fc640 10 monclient: dump: 2024-09-13T11:53:11.339 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:11.339 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:11.339 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.340 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.340 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:11.340 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:11.340 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:11.340 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:11.340 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:11.340 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:11.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.243+0000 7f85d1575640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:11.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.243+0000 7f85ca7fc640 1 -- 172.21.15.100:0/406195307 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85bc0052d0 con 0x7f85b8003a50 2024-09-13T11:53:11.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.243+0000 7f85ca7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:11.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.243+0000 7f85d1575640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:11.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.243+0000 7f85d1575640 4 set_mon_vals no callback set 2024-09-13T11:53:11.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.243+0000 7f85ca7fc640 1 -- 172.21.15.100:0/406195307 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85bc02aa80 con 0x7f85b8003a50 2024-09-13T11:53:11.340 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.243+0000 7f85ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:11.341 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.243+0000 7f85ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:11.341 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.243+0000 7f85ca7fc640 10 monclient: dump: 2024-09-13T11:53:11.341 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:11.341 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:11.341 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.341 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.341 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:11.341 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:11.341 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:11.341 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:11.341 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:11.341 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:11.341 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.243+0000 7f85d1575640 10 monclient: shutdown 2024-09-13T11:53:11.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.243+0000 7f85d1575640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:11.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.243+0000 7f85d1575640 1 -- 172.21.15.100:0/406195307 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85b8003a50 msgr2=0x7f85b8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:11.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.243+0000 7f85d1575640 1 --2- 172.21.15.100:0/406195307 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85b8003a50 0x7f85b8003e70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f85bc001790 tx=0x7f85b8005830 comp rx=0 tx=0).stop 2024-09-13T11:53:11.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.243+0000 7f85d1575640 1 -- 172.21.15.100:0/406195307 shutdown_connections 2024-09-13T11:53:11.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.243+0000 7f85d1575640 1 --2- 172.21.15.100:0/406195307 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85b8003a50 0x7f85b8003e70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.243+0000 7f85d1575640 1 -- 172.21.15.100:0/406195307 >> 172.21.15.100:0/406195307 conn(0x7f85cc0a6c30 msgr2=0x7f85cc0a7040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:11.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.243+0000 7f85d1575640 1 -- 172.21.15.100:0/406195307 shutdown_connections 2024-09-13T11:53:11.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.243+0000 7f85d1575640 1 -- 172.21.15.100:0/406195307 wait complete. 2024-09-13T11:53:11.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.244+0000 7f85d1575640 10 monclient: build_initial_monmap 2024-09-13T11:53:11.342 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.244+0000 7f85d1575640 10 monclient: monmap: 2024-09-13T11:53:11.342 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:11.342 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:11.342 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:11.343 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:11.343 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:11.343 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:11.343 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:11.343 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:11.343 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:11.343 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:11.343 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.244+0000 7f85d1575640 1 Processor -- start 2024-09-13T11:53:11.343 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.244+0000 7f85d1575640 1 -- start start 2024-09-13T11:53:11.343 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.244+0000 7f85d1575640 10 monclient: init 2024-09-13T11:53:11.343 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85d1575640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:11.343 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85d1575640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:53:11.343 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85d1575640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85b8003a50 0x7f85cc13d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:11.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85d1575640 10 monclient(hunting): picked mon.noname-b con 0x7f85b8003a50 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:11.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85d1575640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85cc13dbf0 0x7f85cc1cc820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:11.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85d1575640 10 monclient(hunting): picked mon.noname-c con 0x7f85cc13dbf0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:11.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85d1575640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85cc1ccd60 0x7f85cc1c92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:11.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85d1575640 10 monclient(hunting): picked mon.noname-a con 0x7f85cc1ccd60 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:11.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85d1575640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:11.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85d1575640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc0a36c0 con 0x7f85cc1ccd60 2024-09-13T11:53:11.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85d1575640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:11.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85d1575640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc0a4450 con 0x7f85b8003a50 2024-09-13T11:53:11.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85d1575640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:11.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85d1575640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f85cc0a3840 con 0x7f85cc13dbf0 2024-09-13T11:53:11.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85d1575640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:11.344 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85d1575640 10 monclient(hunting): authenticate will time out at 771.839240s 2024-09-13T11:53:11.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c27fc640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85cc13dbf0 0x7f85cc1cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:11.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c27fc640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85cc13dbf0 0x7f85cc1cc820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47580/0 (socket says 172.21.15.100:47580) 2024-09-13T11:53:11.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c27fc640 1 -- 172.21.15.100:0/2285438242 learned_addr learned my addr 172.21.15.100:0/2285438242 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:11.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c27fc640 10 monclient(hunting): get_auth_request con 0x7f85cc13dbf0 auth_method 0 2024-09-13T11:53:11.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:11.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c27fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:11.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:11.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85cb7fe640 1 --2- 172.21.15.100:0/2285438242 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85cc1ccd60 0x7f85cc1c92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:11.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85cb7fe640 10 monclient(hunting): get_auth_request con 0x7f85cc1ccd60 auth_method 0 2024-09-13T11:53:11.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:11.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:11.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:11.345 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:11.346 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:11.346 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:11.346 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85caffd640 1 --2- 172.21.15.100:0/2285438242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85b8003a50 0x7f85cc13d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:11.346 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:11.348 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:11.348 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:11.348 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85caffd640 10 monclient(hunting): get_auth_request con 0x7f85b8003a50 auth_method 0 2024-09-13T11:53:11.348 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:11.348 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85caffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:11.348 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:11.348 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c27fc640 10 monclient(hunting): handle_auth_done global_id 4352 payload 824 2024-09-13T11:53:11.348 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c27fc640 1 -- 172.21.15.100:0/2285438242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85b8003a50 msgr2=0x7f85cc13d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:11.348 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c27fc640 1 --2- 172.21.15.100:0/2285438242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85b8003a50 0x7f85cc13d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c27fc640 1 -- 172.21.15.100:0/2285438242 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85cc1ccd60 msgr2=0x7f85cc1c92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:11.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c27fc640 1 --2- 172.21.15.100:0/2285438242 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85cc1ccd60 0x7f85cc1c92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c27fc640 10 monclient: _finish_hunting 0 2024-09-13T11:53:11.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c27fc640 1 monclient: found mon.noname-c 2024-09-13T11:53:11.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:11.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:53:11.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c27fc640 1 -- 172.21.15.100:0/2285438242 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85cc0a1140 con 0x7f85cc13dbf0 2024-09-13T11:53:11.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85caffd640 1 --2- 172.21.15.100:0/2285438242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85b8003a50 0x7f85cc13d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:53:11.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85cb7fe640 1 --2- 172.21.15.100:0/2285438242 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85cc1ccd60 0x7f85cc1c92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:11.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c27fc640 1 --2- 172.21.15.100:0/2285438242 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85cc13dbf0 0x7f85cc1cc820 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f85b4007340 tx=0x7f85b4007740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:11.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c3fff640 1 -- 172.21.15.100:0/2285438242 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b40016c0 con 0x7f85cc13dbf0 2024-09-13T11:53:11.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:11.350 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:11.350 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c3fff640 10 monclient: dump: 2024-09-13T11:53:11.350 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:11.350 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:11.350 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.350 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.350 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:11.350 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:11.350 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:11.350 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:11.350 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:11.350 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:11.350 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c3fff640 10 monclient: _finish_auth 0 2024-09-13T11:53:11.350 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c3fff640 10 monclient: _check_auth_tickets 2024-09-13T11:53:11.351 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:11.351 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c3fff640 1 -- 172.21.15.100:0/2285438242 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85b4018070 con 0x7f85cc13dbf0 2024-09-13T11:53:11.351 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85d1575640 5 monclient: authenticate success, global_id 4352 2024-09-13T11:53:11.351 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85d1575640 10 monclient: _renew_subs 2024-09-13T11:53:11.351 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85d1575640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:11.351 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85d1575640 1 -- 172.21.15.100:0/2285438242 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85cc126cf0 con 0x7f85cc13dbf0 2024-09-13T11:53:11.351 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c3fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:11.351 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c3fff640 1 -- 172.21.15.100:0/2285438242 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b400fb20 con 0x7f85cc13dbf0 2024-09-13T11:53:11.351 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:11.351 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:11.351 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c3fff640 10 monclient: dump: 2024-09-13T11:53:11.351 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:11.351 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:11.352 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.352 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.352 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:11.352 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:11.352 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:11.352 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:11.352 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:11.352 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:11.352 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85c9ffb640 4 set_mon_vals no callback set 2024-09-13T11:53:11.352 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85d1575640 10 monclient: _renew_subs 2024-09-13T11:53:11.352 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85d1575640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:11.352 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.245+0000 7f85d1575640 1 -- 172.21.15.100:0/2285438242 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f85cc1287b0 con 0x7f85cc13dbf0 2024-09-13T11:53:11.352 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.246+0000 7f85c3fff640 1 -- 172.21.15.100:0/2285438242 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f85b400fcc0 con 0x7f85cc13dbf0 2024-09-13T11:53:11.352 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.246+0000 7f85d1575640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:11.353 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.246+0000 7f85d1575640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:11.353 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.246+0000 7f85d1575640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:11.353 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.246+0000 7f85d1575640 1 -- 172.21.15.100:0/2285438242 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85cc0a3c20 con 0x7f85cc13dbf0 2024-09-13T11:53:11.353 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.247+0000 7f85c3fff640 1 --2- 172.21.15.100:0/2285438242 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f85ac03dfa0 0x7f85ac040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:11.353 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.247+0000 7f85c3fff640 1 -- 172.21.15.100:0/2285438242 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4374+0+0 (secure 0 0 0) 0x7f85b40523f0 con 0x7f85cc13dbf0 2024-09-13T11:53:11.353 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.247+0000 7f85caffd640 1 --2- 172.21.15.100:0/2285438242 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f85ac03dfa0 0x7f85ac040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:11.353 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.247+0000 7f85caffd640 10 monclient: get_auth_request con 0x7f85ac03dfa0 auth_method 0 2024-09-13T11:53:11.353 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.247+0000 7f85caffd640 1 --2- 172.21.15.100:0/2285438242 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f85ac03dfa0 0x7f85ac040460 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f85bc032000 tx=0x7f85bc02e000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:11.353 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.251+0000 7f85c3fff640 1 -- 172.21.15.100:0/2285438242 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f85b4012d60 con 0x7f85cc13dbf0 2024-09-13T11:53:11.353 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.251+0000 7f85c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:11.353 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.251+0000 7f85c3fff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:11.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.426+0000 7f85d1575640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-13T11:53:11.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.426+0000 7f85d1575640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-13T11:53:11.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.426+0000 7f85d1575640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:11.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.426+0000 7f85d1575640 1 -- 172.21.15.100:0/2285438242 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f85cc0a5b60 con 0x7f85cc13dbf0 2024-09-13T11:53:11.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.426+0000 7f85c3fff640 1 -- 172.21.15.100:0/2285438242 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) ==== 93+0+10 (secure 0 0 0) 0x7f85b4052090 con 0x7f85cc13dbf0 2024-09-13T11:53:11.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.426+0000 7f85c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-13T11:53:11.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.426+0000 7f85c3fff640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:11.429 INFO:teuthology.orchestra.run.smithi100.stdout:pg_num: 1 2024-09-13T11:53:11.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.428+0000 7f85d1575640 1 -- 172.21.15.100:0/2285438242 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f85ac03dfa0 msgr2=0x7f85ac040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:11.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.428+0000 7f85d1575640 1 --2- 172.21.15.100:0/2285438242 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f85ac03dfa0 0x7f85ac040460 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f85bc032000 tx=0x7f85bc02e000 comp rx=0 tx=0).stop 2024-09-13T11:53:11.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.428+0000 7f85d1575640 10 monclient: shutdown 2024-09-13T11:53:11.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.428+0000 7f85d1575640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:11.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.428+0000 7f85d1575640 1 -- 172.21.15.100:0/2285438242 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85cc13dbf0 msgr2=0x7f85cc1cc820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:11.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.428+0000 7f85d1575640 1 --2- 172.21.15.100:0/2285438242 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85cc13dbf0 0x7f85cc1cc820 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f85b4007340 tx=0x7f85b4007740 comp rx=0 tx=0).stop 2024-09-13T11:53:11.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.428+0000 7f85d1575640 1 -- 172.21.15.100:0/2285438242 shutdown_connections 2024-09-13T11:53:11.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.428+0000 7f85d1575640 1 --2- 172.21.15.100:0/2285438242 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f85ac03dfa0 0x7f85ac040460 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.428+0000 7f85d1575640 1 --2- 172.21.15.100:0/2285438242 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85cc1ccd60 0x7f85cc1c92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.428+0000 7f85d1575640 1 --2- 172.21.15.100:0/2285438242 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85cc13dbf0 0x7f85cc1cc820 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.428+0000 7f85d1575640 1 --2- 172.21.15.100:0/2285438242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85b8003a50 0x7f85cc13d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.428+0000 7f85d1575640 1 -- 172.21.15.100:0/2285438242 >> 172.21.15.100:0/2285438242 conn(0x7f85cc0a6c30 msgr2=0x7f85cc0a8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:11.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.429+0000 7f85d1575640 1 -- 172.21.15.100:0/2285438242 shutdown_connections 2024-09-13T11:53:11.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.429+0000 7f85d1575640 1 -- 172.21.15.100:0/2285438242 wait complete. 2024-09-13T11:53:11.441 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-13T11:53:11.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.539+0000 7f95a6f98640 10 monclient: get_monmap_and_config 2024-09-13T11:53:11.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.539+0000 7f95a6f98640 10 monclient: build_initial_monmap 2024-09-13T11:53:11.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.539+0000 7f95a6f98640 10 monclient: monmap: 2024-09-13T11:53:11.542 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:11.542 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:11.542 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:11.541338+0000 2024-09-13T11:53:11.542 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:11.541338+0000 2024-09-13T11:53:11.542 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:11.542 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:11.542 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:11.542 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:11.542 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:11.542 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:11.543 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:11.543 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:11.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f95a6f98640 1 Processor -- start 2024-09-13T11:53:11.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f95a6f98640 1 -- start start 2024-09-13T11:53:11.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f95a6f98640 10 monclient: init 2024-09-13T11:53:11.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f95a6f98640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:11.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f95a6f98640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-13T11:53:11.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f95a6f98640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f95a00a33f0 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:11.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f95a6f98640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95a00a37d0 0x7f95a013c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:11.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f95a6f98640 10 monclient(hunting): picked mon.noname-b con 0x7f95a00a37d0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:11.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f95a6f98640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f95a013cc10 0x7f95a013f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:11.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f95a6f98640 10 monclient(hunting): picked mon.noname-a con 0x7f95a013cc10 addr v2:172.21.15.100:3300/0 2024-09-13T11:53:11.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f95a6f98640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95a00a3bb0 con 0x7f95a00a33f0 2024-09-13T11:53:11.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f95a6f98640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:11.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f95a6f98640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f95a00a1d40 con 0x7f95a013cc10 2024-09-13T11:53:11.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f95a6f98640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:11.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f95a6f98640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f95a00a9800 con 0x7f95a00a37d0 2024-09-13T11:53:11.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f95a6f98640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:11.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f95a6f98640 10 monclient(hunting): authenticate will time out at 772.134770s 2024-09-13T11:53:11.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f95a5535640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f95a013cc10 0x7f95a013f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:11.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f959ffff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95a00a37d0 0x7f95a013c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:11.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f95a5535640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f95a013cc10 0x7f95a013f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49772/0 (socket says 172.21.15.100:49772) 2024-09-13T11:53:11.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f959ffff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95a00a37d0 0x7f95a013c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47588/0 (socket says 172.21.15.100:47588) 2024-09-13T11:53:11.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f95a5535640 1 -- 172.21.15.100:0/3892209956 learned_addr learned my addr 172.21.15.100:0/3892209956 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:11.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f95a5535640 10 monclient(hunting): get_auth_request con 0x7f95a013cc10 auth_method 0 2024-09-13T11:53:11.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f95a5535640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:11.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f95a5535640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:11.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f95a5535640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:11.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f959ffff640 10 monclient(hunting): get_auth_request con 0x7f95a00a37d0 auth_method 0 2024-09-13T11:53:11.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f959ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:11.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f959ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:11.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.540+0000 7f959ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:11.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.541+0000 7f959ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:11.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.541+0000 7f959ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:11.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.541+0000 7f959ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:11.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.541+0000 7f95a5535640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:11.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.541+0000 7f95a5535640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:11.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.541+0000 7f95a5535640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:11.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.541+0000 7f959ffff640 10 monclient(hunting): handle_auth_done global_id 4355 payload 274 2024-09-13T11:53:11.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.541+0000 7f959ffff640 1 -- 172.21.15.100:0/3892209956 >> v2:172.21.15.100:3300/0 conn(0x7f95a013cc10 msgr2=0x7f95a013f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:11.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.541+0000 7f959ffff640 1 --2- 172.21.15.100:0/3892209956 >> v2:172.21.15.100:3300/0 conn(0x7f95a013cc10 0x7f95a013f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.541+0000 7f959ffff640 1 -- 172.21.15.100:0/3892209956 >> v1:172.21.15.122:6789/0 conn(0x7f95a00a33f0 legacy=0x7f95a00a26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:11.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.541+0000 7f959ffff640 10 monclient: _finish_hunting 0 2024-09-13T11:53:11.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.541+0000 7f959ffff640 1 monclient: found mon.noname-b 2024-09-13T11:53:11.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.541+0000 7f959ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:11.557 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.541+0000 7f959ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:53:11.557 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.541+0000 7f959ffff640 1 -- 172.21.15.100:0/3892209956 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95a00a14a0 con 0x7f95a00a37d0 2024-09-13T11:53:11.557 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.541+0000 7f95a5535640 1 --2- 172.21.15.100:0/3892209956 >> v2:172.21.15.100:3300/0 conn(0x7f95a013cc10 0x7f95a013f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:11.557 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.541+0000 7f959ffff640 1 --2- 172.21.15.100:0/3892209956 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95a00a37d0 0x7f95a013c6d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f958c0097a0 tx=0x7f958c029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72147072ef751781 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:11.557 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.541+0000 7f959f7fe640 1 -- 172.21.15.100:0/3892209956 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958c037070 con 0x7f95a00a37d0 2024-09-13T11:53:11.557 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.541+0000 7f959f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:11.557 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.541+0000 7f959f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:11.557 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.541+0000 7f959f7fe640 10 monclient: dump: 2024-09-13T11:53:11.557 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:11.557 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:11.557 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.557 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.558 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:11.558 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:11.558 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:11.558 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:11.558 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:11.558 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:11.558 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.542+0000 7f959f7fe640 10 monclient: _finish_auth 0 2024-09-13T11:53:11.558 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.542+0000 7f959f7fe640 10 monclient: _check_auth_tickets 2024-09-13T11:53:11.558 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.542+0000 7f959f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:11.558 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.542+0000 7f959f7fe640 1 -- 172.21.15.100:0/3892209956 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f958c02ad20 con 0x7f95a00a37d0 2024-09-13T11:53:11.558 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.542+0000 7f95a6f98640 5 monclient: authenticate success, global_id 4355 2024-09-13T11:53:11.558 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.542+0000 7f95a6f98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:11.558 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.542+0000 7f959f7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:11.559 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.542+0000 7f95a6f98640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:11.559 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.542+0000 7f959f7fe640 1 -- 172.21.15.100:0/3892209956 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958c032660 con 0x7f95a00a37d0 2024-09-13T11:53:11.559 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.542+0000 7f959f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:11.559 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.542+0000 7f95a6f98640 4 set_mon_vals no callback set 2024-09-13T11:53:11.559 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.542+0000 7f959f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:11.559 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.542+0000 7f959f7fe640 10 monclient: dump: 2024-09-13T11:53:11.559 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:11.559 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:11.559 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.559 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.559 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:11.559 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:11.560 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:11.560 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:11.560 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:11.560 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:11.560 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.542+0000 7f95a6f98640 10 monclient: shutdown 2024-09-13T11:53:11.560 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.542+0000 7f95a6f98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:11.560 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.542+0000 7f95a6f98640 1 -- 172.21.15.100:0/3892209956 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95a00a37d0 msgr2=0x7f95a013c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:11.560 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.542+0000 7f95a6f98640 1 --2- 172.21.15.100:0/3892209956 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95a00a37d0 0x7f95a013c6d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f958c0097a0 tx=0x7f958c029fd0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.560 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.542+0000 7f95a6f98640 1 -- 172.21.15.100:0/3892209956 shutdown_connections 2024-09-13T11:53:11.560 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.542+0000 7f95a6f98640 1 --2- 172.21.15.100:0/3892209956 >> v2:172.21.15.100:3300/0 conn(0x7f95a013cc10 0x7f95a013f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.560 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.542+0000 7f95a6f98640 1 --2- 172.21.15.100:0/3892209956 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95a00a37d0 0x7f95a013c6d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.560 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.542+0000 7f95a6f98640 1 -- 172.21.15.100:0/3892209956 >> 172.21.15.100:0/3892209956 conn(0x7f95a00a69e0 msgr2=0x7f95a00a6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:11.560 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.542+0000 7f95a6f98640 1 -- 172.21.15.100:0/3892209956 shutdown_connections 2024-09-13T11:53:11.561 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.542+0000 7f95a6f98640 1 -- 172.21.15.100:0/3892209956 wait complete. 2024-09-13T11:53:11.561 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.543+0000 7f95a6f98640 10 monclient: build_initial_monmap 2024-09-13T11:53:11.561 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.543+0000 7f95a6f98640 10 monclient: monmap: 2024-09-13T11:53:11.561 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:11.561 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:11.561 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:11.561 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:11.561 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:11.561 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:11.561 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:11.561 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:11.561 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:11.561 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:11.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.543+0000 7f95a6f98640 1 Processor -- start 2024-09-13T11:53:11.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.543+0000 7f95a6f98640 1 -- start start 2024-09-13T11:53:11.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.543+0000 7f95a6f98640 10 monclient: init 2024-09-13T11:53:11.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a6f98640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:11.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a6f98640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:53:11.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a6f98640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f95a00a33f0 0x7f95a0134a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:11.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a6f98640 10 monclient(hunting): picked mon.noname-a con 0x7f95a00a33f0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:11.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a6f98640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95a00a37d0 0x7f95a0134f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:11.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a6f98640 10 monclient(hunting): picked mon.noname-b con 0x7f95a00a37d0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:11.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a6f98640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95a013cc10 0x7f95a01ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:11.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a6f98640 10 monclient(hunting): picked mon.noname-c con 0x7f95a013cc10 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:11.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a6f98640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:11.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a6f98640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f95a013f540 con 0x7f95a00a33f0 2024-09-13T11:53:11.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a6f98640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:11.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a6f98640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f95a013f6c0 con 0x7f95a00a37d0 2024-09-13T11:53:11.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a6f98640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:11.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a6f98640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f95a013f840 con 0x7f95a013cc10 2024-09-13T11:53:11.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a6f98640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:11.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a6f98640 10 monclient(hunting): authenticate will time out at 772.138156s 2024-09-13T11:53:11.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a4d34640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f95a00a33f0 0x7f95a0134a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:11.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a4d34640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f95a00a33f0 0x7f95a0134a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49776/0 (socket says 172.21.15.100:49776) 2024-09-13T11:53:11.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a4d34640 1 -- 172.21.15.100:0/1737858710 learned_addr learned my addr 172.21.15.100:0/1737858710 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:11.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a5535640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95a013cc10 0x7f95a01ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:11.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a4d34640 10 monclient(hunting): get_auth_request con 0x7f95a00a33f0 auth_method 0 2024-09-13T11:53:11.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a4d34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:11.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a4d34640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:11.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a4d34640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:11.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a5535640 10 monclient(hunting): get_auth_request con 0x7f95a013cc10 auth_method 0 2024-09-13T11:53:11.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a5535640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:11.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a5535640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:11.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a5535640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:11.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a4d34640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:11.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a4d34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:11.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a4d34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:11.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a5535640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:11.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a5535640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:11.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a5535640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:11.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f959ffff640 1 --2- 172.21.15.100:0/1737858710 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95a00a37d0 0x7f95a0134f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:11.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f959ffff640 10 monclient(hunting): get_auth_request con 0x7f95a00a37d0 auth_method 0 2024-09-13T11:53:11.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f959ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:11.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f959ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:11.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f959ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:11.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a4d34640 10 monclient(hunting): handle_auth_done global_id 4395 payload 824 2024-09-13T11:53:11.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a4d34640 1 -- 172.21.15.100:0/1737858710 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95a013cc10 msgr2=0x7f95a01ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:11.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a4d34640 1 --2- 172.21.15.100:0/1737858710 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95a013cc10 0x7f95a01ccc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a4d34640 1 -- 172.21.15.100:0/1737858710 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95a00a37d0 msgr2=0x7f95a0134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:11.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a4d34640 1 --2- 172.21.15.100:0/1737858710 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95a00a37d0 0x7f95a0134f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a4d34640 10 monclient: _finish_hunting 0 2024-09-13T11:53:11.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a4d34640 1 monclient: found mon.noname-a 2024-09-13T11:53:11.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a4d34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:11.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a4d34640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:11.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a4d34640 1 -- 172.21.15.100:0/1737858710 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95a00a10d0 con 0x7f95a00a33f0 2024-09-13T11:53:11.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a4d34640 1 --2- 172.21.15.100:0/1737858710 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f95a00a33f0 0x7f95a0134a50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f959400c430 tx=0x7f959400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:11.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f959d7fa640 1 -- 172.21.15.100:0/1737858710 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9594011790 con 0x7f95a00a33f0 2024-09-13T11:53:11.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f959d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:11.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f959d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:11.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f959d7fa640 10 monclient: dump: 2024-09-13T11:53:11.566 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:11.566 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:11.566 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.566 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.566 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:11.566 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:11.566 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:11.566 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:11.566 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:11.567 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:11.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f959d7fa640 10 monclient: _finish_auth 0 2024-09-13T11:53:11.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f959d7fa640 10 monclient: _check_auth_tickets 2024-09-13T11:53:11.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f959d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:11.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f959d7fa640 1 -- 172.21.15.100:0/1737858710 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9594017070 con 0x7f95a00a33f0 2024-09-13T11:53:11.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a6f98640 5 monclient: authenticate success, global_id 4395 2024-09-13T11:53:11.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a6f98640 10 monclient: _renew_subs 2024-09-13T11:53:11.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a6f98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:11.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a6f98640 1 -- 172.21.15.100:0/1737858710 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95a0070120 con 0x7f95a00a33f0 2024-09-13T11:53:11.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f959d7fa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:11.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f959d7fa640 1 -- 172.21.15.100:0/1737858710 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9594011bb0 con 0x7f95a00a33f0 2024-09-13T11:53:11.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f959d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:11.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f959d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:11.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f959d7fa640 10 monclient: dump: 2024-09-13T11:53:11.568 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:11.568 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:11.568 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.568 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.568 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:11.568 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:11.568 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:11.568 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:11.568 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:11.568 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:11.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f959effd640 4 set_mon_vals no callback set 2024-09-13T11:53:11.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a6f98640 10 monclient: _renew_subs 2024-09-13T11:53:11.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a6f98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:11.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.544+0000 7f95a6f98640 1 -- 172.21.15.100:0/1737858710 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95a013b3f0 con 0x7f95a00a33f0 2024-09-13T11:53:11.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.545+0000 7f95a6f98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:11.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.545+0000 7f95a6f98640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:11.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.545+0000 7f95a6f98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:11.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.545+0000 7f95a6f98640 1 -- 172.21.15.100:0/1737858710 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95a00a26f0 con 0x7f95a00a33f0 2024-09-13T11:53:11.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.545+0000 7f959d7fa640 1 -- 172.21.15.100:0/1737858710 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f959401c460 con 0x7f95a00a33f0 2024-09-13T11:53:11.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.546+0000 7f959d7fa640 1 --2- 172.21.15.100:0/1737858710 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f958403dfa0 0x7f9584040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:11.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.546+0000 7f959d7fa640 1 -- 172.21.15.100:0/1737858710 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4374+0+0 (secure 0 0 0) 0x7f95940523b0 con 0x7f95a00a33f0 2024-09-13T11:53:11.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.546+0000 7f959ffff640 1 --2- 172.21.15.100:0/1737858710 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f958403dfa0 0x7f9584040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:11.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.546+0000 7f959ffff640 10 monclient: get_auth_request con 0x7f958403dfa0 auth_method 0 2024-09-13T11:53:11.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.547+0000 7f959ffff640 1 --2- 172.21.15.100:0/1737858710 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f958403dfa0 0x7f9584040460 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f958c034000 tx=0x7f958c02e000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:11.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.550+0000 7f959d7fa640 1 -- 172.21.15.100:0/1737858710 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f95a00a10d0 con 0x7f95a00a33f0 2024-09-13T11:53:11.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.550+0000 7f959d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:11.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.550+0000 7f959d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:11.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.724+0000 7f95a6f98640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-13T11:53:11.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.724+0000 7f95a6f98640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-13T11:53:11.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.724+0000 7f95a6f98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:11.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.724+0000 7f95a6f98640 1 -- 172.21.15.100:0/1737858710 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7f95a00a59a0 con 0x7f95a00a33f0 2024-09-13T11:53:11.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.724+0000 7f959d7fa640 1 -- 172.21.15.100:0/1737858710 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) ==== 92+0+10 (secure 0 0 0) 0x7f95940520d0 con 0x7f95a00a33f0 2024-09-13T11:53:11.728 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.724+0000 7f959d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-13T11:53:11.728 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.724+0000 7f959d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:11.728 INFO:teuthology.orchestra.run.smithi100.stdout:pg_num: 8 2024-09-13T11:53:11.729 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.727+0000 7f95a6f98640 1 -- 172.21.15.100:0/1737858710 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f958403dfa0 msgr2=0x7f9584040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:11.729 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.727+0000 7f95a6f98640 1 --2- 172.21.15.100:0/1737858710 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f958403dfa0 0x7f9584040460 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f958c034000 tx=0x7f958c02e000 comp rx=0 tx=0).stop 2024-09-13T11:53:11.730 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.727+0000 7f95a6f98640 10 monclient: shutdown 2024-09-13T11:53:11.730 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.727+0000 7f95a6f98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:11.730 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.727+0000 7f95a6f98640 1 -- 172.21.15.100:0/1737858710 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f95a00a33f0 msgr2=0x7f95a0134a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:11.730 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.727+0000 7f95a6f98640 1 --2- 172.21.15.100:0/1737858710 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f95a00a33f0 0x7f95a0134a50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f959400c430 tx=0x7f959400c8f0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.730 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.727+0000 7f95a6f98640 1 -- 172.21.15.100:0/1737858710 shutdown_connections 2024-09-13T11:53:11.730 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.727+0000 7f95a6f98640 1 --2- 172.21.15.100:0/1737858710 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f958403dfa0 0x7f9584040460 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.730 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.727+0000 7f95a6f98640 1 --2- 172.21.15.100:0/1737858710 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95a013cc10 0x7f95a01ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.730 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.727+0000 7f95a6f98640 1 --2- 172.21.15.100:0/1737858710 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95a00a37d0 0x7f95a0134f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.730 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.727+0000 7f95a6f98640 1 --2- 172.21.15.100:0/1737858710 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f95a00a33f0 0x7f95a0134a50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.730 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.727+0000 7f95a6f98640 1 -- 172.21.15.100:0/1737858710 >> 172.21.15.100:0/1737858710 conn(0x7f95a00a69e0 msgr2=0x7f95a00a5f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:11.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.727+0000 7f95a6f98640 1 -- 172.21.15.100:0/1737858710 shutdown_connections 2024-09-13T11:53:11.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.727+0000 7f95a6f98640 1 -- 172.21.15.100:0/1737858710 wait complete. 2024-09-13T11:53:11.740 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-13T11:53:11.740 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:53:11.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.834+0000 7f25912b6640 10 monclient: get_monmap_and_config 2024-09-13T11:53:11.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.834+0000 7f25912b6640 10 monclient: build_initial_monmap 2024-09-13T11:53:11.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.834+0000 7f25912b6640 10 monclient: monmap: 2024-09-13T11:53:11.837 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:11.837 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:11.837 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:11.836336+0000 2024-09-13T11:53:11.838 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:11.836336+0000 2024-09-13T11:53:11.838 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:11.838 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:11.838 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:11.838 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:11.838 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:11.838 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:11.838 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:11.838 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:11.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f25912b6640 1 Processor -- start 2024-09-13T11:53:11.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f25912b6640 1 -- start start 2024-09-13T11:53:11.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f25912b6640 10 monclient: init 2024-09-13T11:53:11.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f25912b6640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:11.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f25912b6640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-13T11:53:11.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f25912b6640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f258c137a60 0x7f258c0a2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:11.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f25912b6640 10 monclient(hunting): picked mon.noname-b con 0x7f258c137a60 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:11.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f25912b6640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f258c137e40 0x7f258c0a4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:11.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f25912b6640 10 monclient(hunting): picked mon.noname-c con 0x7f258c137e40 addr v2:172.21.15.122:3300/0 2024-09-13T11:53:11.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f25912b6640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f258c138240 0x7f258c0a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:11.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f25912b6640 10 monclient(hunting): picked mon.noname-a con 0x7f258c138240 addr v2:172.21.15.100:3300/0 2024-09-13T11:53:11.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f25912b6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:11.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f25912b6640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f258c0a4410 con 0x7f258c138240 2024-09-13T11:53:11.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f25912b6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:11.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f25912b6640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f258c0a5be0 con 0x7f258c137e40 2024-09-13T11:53:11.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f25912b6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:11.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f25912b6640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f258c0a76b0 con 0x7f258c137a60 2024-09-13T11:53:11.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f25912b6640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:11.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f25912b6640 10 monclient(hunting): authenticate will time out at 772.429576s 2024-09-13T11:53:11.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258b7fe640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f258c138240 0x7f258c0a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:11.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258affd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f258c137a60 0x7f258c0a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:11.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258b7fe640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f258c138240 0x7f258c0a4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49784/0 (socket says 172.21.15.100:49784) 2024-09-13T11:53:11.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258b7fe640 1 -- 172.21.15.100:0/2504003994 learned_addr learned my addr 172.21.15.100:0/2504003994 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:11.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258affd640 10 monclient(hunting): get_auth_request con 0x7f258c137a60 auth_method 0 2024-09-13T11:53:11.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:11.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:11.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:11.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258b7fe640 10 monclient(hunting): get_auth_request con 0x7f258c138240 auth_method 0 2024-09-13T11:53:11.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:11.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:11.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:11.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:11.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:11.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:11.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:11.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:11.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258a7fc640 1 --2- 172.21.15.100:0/2504003994 >> v2:172.21.15.122:3300/0 conn(0x7f258c137e40 0x7f258c0a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:11.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:11.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258a7fc640 10 monclient(hunting): get_auth_request con 0x7f258c137e40 auth_method 0 2024-09-13T11:53:11.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:11.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:11.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:11.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258b7fe640 10 monclient(hunting): handle_auth_done global_id 4398 payload 274 2024-09-13T11:53:11.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258b7fe640 1 -- 172.21.15.100:0/2504003994 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f258c137a60 msgr2=0x7f258c0a2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:11.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258b7fe640 1 --2- 172.21.15.100:0/2504003994 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f258c137a60 0x7f258c0a2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258b7fe640 1 -- 172.21.15.100:0/2504003994 >> v2:172.21.15.122:3300/0 conn(0x7f258c137e40 msgr2=0x7f258c0a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:11.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258b7fe640 1 --2- 172.21.15.100:0/2504003994 >> v2:172.21.15.122:3300/0 conn(0x7f258c137e40 0x7f258c0a4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258b7fe640 10 monclient: _finish_hunting 0 2024-09-13T11:53:11.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258b7fe640 1 monclient: found mon.noname-a 2024-09-13T11:53:11.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:11.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:11.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258b7fe640 1 -- 172.21.15.100:0/2504003994 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f258c0a19e0 con 0x7f258c138240 2024-09-13T11:53:11.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258affd640 1 --2- 172.21.15.100:0/2504003994 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f258c137a60 0x7f258c0a2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:11.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.835+0000 7f258a7fc640 1 --2- 172.21.15.100:0/2504003994 >> v2:172.21.15.122:3300/0 conn(0x7f258c137e40 0x7f258c0a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:53:11.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.836+0000 7f258b7fe640 1 --2- 172.21.15.100:0/2504003994 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f258c138240 0x7f258c0a4e90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f258c070390 tx=0x7f257c02a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d7797df374e011 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:11.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.836+0000 7f2589ffb640 1 -- 172.21.15.100:0/2504003994 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257c037070 con 0x7f258c138240 2024-09-13T11:53:11.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.836+0000 7f2589ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:11.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.836+0000 7f2589ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:11.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.836+0000 7f2589ffb640 10 monclient: dump: 2024-09-13T11:53:11.852 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:11.852 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:11.852 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.852 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.852 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:11.852 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:11.853 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:11.853 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:11.853 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:11.853 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:11.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.836+0000 7f2589ffb640 10 monclient: _finish_auth 0 2024-09-13T11:53:11.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.836+0000 7f2589ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:53:11.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.836+0000 7f2589ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:11.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.836+0000 7f2589ffb640 1 -- 172.21.15.100:0/2504003994 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f257c02ae00 con 0x7f258c138240 2024-09-13T11:53:11.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.836+0000 7f2589ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:11.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.836+0000 7f2589ffb640 1 -- 172.21.15.100:0/2504003994 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257c032660 con 0x7f258c138240 2024-09-13T11:53:11.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.836+0000 7f25912b6640 5 monclient: authenticate success, global_id 4398 2024-09-13T11:53:11.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.836+0000 7f25912b6640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:11.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.836+0000 7f25912b6640 4 set_mon_vals no callback set 2024-09-13T11:53:11.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.836+0000 7f2589ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:11.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.836+0000 7f2589ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:11.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.836+0000 7f2589ffb640 10 monclient: dump: 2024-09-13T11:53:11.854 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:11.854 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:11.854 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.854 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.854 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:11.854 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:11.854 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:11.855 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:11.855 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:11.855 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:11.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.837+0000 7f25912b6640 10 monclient: shutdown 2024-09-13T11:53:11.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.837+0000 7f25912b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:11.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.837+0000 7f25912b6640 1 -- 172.21.15.100:0/2504003994 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f258c138240 msgr2=0x7f258c0a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:11.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.837+0000 7f25912b6640 1 --2- 172.21.15.100:0/2504003994 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f258c138240 0x7f258c0a4e90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f258c070390 tx=0x7f257c02a0b0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.837+0000 7f25912b6640 1 -- 172.21.15.100:0/2504003994 shutdown_connections 2024-09-13T11:53:11.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.837+0000 7f25912b6640 1 --2- 172.21.15.100:0/2504003994 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f258c138240 0x7f258c0a4e90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.837+0000 7f25912b6640 1 --2- 172.21.15.100:0/2504003994 >> v2:172.21.15.122:3300/0 conn(0x7f258c137e40 0x7f258c0a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.837+0000 7f25912b6640 1 --2- 172.21.15.100:0/2504003994 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f258c137a60 0x7f258c0a2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.837+0000 7f25912b6640 1 -- 172.21.15.100:0/2504003994 >> 172.21.15.100:0/2504003994 conn(0x7f258c0aa580 msgr2=0x7f258c0a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:11.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.837+0000 7f25912b6640 1 -- 172.21.15.100:0/2504003994 shutdown_connections 2024-09-13T11:53:11.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.837+0000 7f25912b6640 1 -- 172.21.15.100:0/2504003994 wait complete. 2024-09-13T11:53:11.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.837+0000 7f25912b6640 10 monclient: build_initial_monmap 2024-09-13T11:53:11.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.838+0000 7f25912b6640 10 monclient: monmap: 2024-09-13T11:53:11.856 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:11.856 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:11.856 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:11.856 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:11.856 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:11.856 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:11.856 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:11.856 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:11.856 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:11.857 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:11.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.838+0000 7f25912b6640 1 Processor -- start 2024-09-13T11:53:11.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.838+0000 7f25912b6640 1 -- start start 2024-09-13T11:53:11.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.838+0000 7f25912b6640 10 monclient: init 2024-09-13T11:53:11.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.838+0000 7f25912b6640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:11.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.838+0000 7f25912b6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:53:11.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.838+0000 7f25912b6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f258c137a60 0x7f258c1cca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:11.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.838+0000 7f25912b6640 10 monclient(hunting): picked mon.noname-b con 0x7f258c137a60 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:11.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.838+0000 7f25912b6640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f258c137e40 0x7f258c1369c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:11.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.838+0000 7f25912b6640 10 monclient(hunting): picked mon.noname-c con 0x7f258c137e40 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:11.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25912b6640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f258c138240 0x7f258c136f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:11.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25912b6640 10 monclient(hunting): picked mon.noname-a con 0x7f258c138240 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:11.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25912b6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:11.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25912b6640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f258c06ff00 con 0x7f258c138240 2024-09-13T11:53:11.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25912b6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:11.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25912b6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f258c06fd80 con 0x7f258c137a60 2024-09-13T11:53:11.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25912b6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:11.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25912b6640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f258c0a53d0 con 0x7f258c137e40 2024-09-13T11:53:11.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25912b6640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:11.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25912b6640 10 monclient(hunting): authenticate will time out at 772.433037s 2024-09-13T11:53:11.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258a7fc640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f258c137e40 0x7f258c1369c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:11.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258b7fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f258c138240 0x7f258c136f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:11.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258a7fc640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f258c137e40 0x7f258c1369c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:47608/0 (socket says 172.21.15.100:47608) 2024-09-13T11:53:11.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258b7fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f258c138240 0x7f258c136f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49792/0 (socket says 172.21.15.100:49792) 2024-09-13T11:53:11.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258a7fc640 1 -- 172.21.15.100:0/1219680756 learned_addr learned my addr 172.21.15.100:0/1219680756 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:11.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258a7fc640 10 monclient(hunting): get_auth_request con 0x7f258c137e40 auth_method 0 2024-09-13T11:53:11.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:11.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:11.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:11.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258b7fe640 10 monclient(hunting): get_auth_request con 0x7f258c138240 auth_method 0 2024-09-13T11:53:11.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:11.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:11.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:11.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:11.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:11.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:11.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:11.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:11.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:11.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258affd640 1 --2- 172.21.15.100:0/1219680756 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f258c137a60 0x7f258c1cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:11.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258b7fe640 10 monclient(hunting): handle_auth_done global_id 4401 payload 824 2024-09-13T11:53:11.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258b7fe640 1 -- 172.21.15.100:0/1219680756 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f258c137e40 msgr2=0x7f258c1369c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:11.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258b7fe640 1 --2- 172.21.15.100:0/1219680756 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f258c137e40 0x7f258c1369c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258b7fe640 1 -- 172.21.15.100:0/1219680756 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f258c137a60 msgr2=0x7f258c1cca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:11.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258b7fe640 1 --2- 172.21.15.100:0/1219680756 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f258c137a60 0x7f258c1cca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:11.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258b7fe640 10 monclient: _finish_hunting 0 2024-09-13T11:53:11.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258b7fe640 1 monclient: found mon.noname-a 2024-09-13T11:53:11.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:11.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:11.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258b7fe640 1 -- 172.21.15.100:0/1219680756 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f258c1283c0 con 0x7f258c138240 2024-09-13T11:53:11.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258affd640 10 monclient: get_auth_request con 0x7f258c137a60 auth_method 0 2024-09-13T11:53:11.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258affd640 1 --2- 172.21.15.100:0/1219680756 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f258c137a60 0x7f258c1cca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:53:11.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f258b7fe640 1 --2- 172.21.15.100:0/1219680756 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f258c138240 0x7f258c136f00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f257c031000 tx=0x7f257c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:11.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25737fe640 1 -- 172.21.15.100:0/1219680756 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257c041070 con 0x7f258c138240 2024-09-13T11:53:11.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:11.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:11.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25737fe640 10 monclient: dump: 2024-09-13T11:53:11.861 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:11.861 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:11.862 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.862 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.862 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:11.862 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:11.862 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:11.862 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:11.862 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:11.862 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:11.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25737fe640 10 monclient: _finish_auth 0 2024-09-13T11:53:11.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25737fe640 10 monclient: _check_auth_tickets 2024-09-13T11:53:11.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:11.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25737fe640 1 -- 172.21.15.100:0/1219680756 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f257c03c460 con 0x7f258c138240 2024-09-13T11:53:11.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25912b6640 5 monclient: authenticate success, global_id 4401 2024-09-13T11:53:11.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25912b6640 10 monclient: _renew_subs 2024-09-13T11:53:11.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25912b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:11.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25912b6640 1 -- 172.21.15.100:0/1219680756 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f258c126bc0 con 0x7f258c138240 2024-09-13T11:53:11.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25737fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:11.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25737fe640 1 -- 172.21.15.100:0/1219680756 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257c037040 con 0x7f258c138240 2024-09-13T11:53:11.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:11.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:11.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25737fe640 10 monclient: dump: 2024-09-13T11:53:11.863 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:11.863 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:11.863 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.863 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:11.864 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:11.864 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:11.864 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:11.864 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:11.864 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:11.864 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:11.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25912b6640 10 monclient: _renew_subs 2024-09-13T11:53:11.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25912b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:11.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25897fa640 4 set_mon_vals no callback set 2024-09-13T11:53:11.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.839+0000 7f25912b6640 1 -- 172.21.15.100:0/1219680756 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f258c137440 con 0x7f258c138240 2024-09-13T11:53:11.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.840+0000 7f25737fe640 1 -- 172.21.15.100:0/1219680756 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f257c03a070 con 0x7f258c138240 2024-09-13T11:53:11.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.840+0000 7f25912b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:11.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.840+0000 7f25912b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:11.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.840+0000 7f25912b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:11.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.840+0000 7f25912b6640 1 -- 172.21.15.100:0/1219680756 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f258c0a2980 con 0x7f258c138240 2024-09-13T11:53:11.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.840+0000 7f25737fe640 1 --2- 172.21.15.100:0/1219680756 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f257803dfa0 0x7f2578040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:11.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.841+0000 7f25737fe640 1 -- 172.21.15.100:0/1219680756 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4374+0+0 (secure 0 0 0) 0x7f257c072850 con 0x7f258c138240 2024-09-13T11:53:11.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.841+0000 7f258affd640 1 --2- 172.21.15.100:0/1219680756 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f257803dfa0 0x7f2578040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:11.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.841+0000 7f258affd640 10 monclient: get_auth_request con 0x7f257803dfa0 auth_method 0 2024-09-13T11:53:11.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.841+0000 7f258affd640 1 --2- 172.21.15.100:0/1219680756 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f257803dfa0 0x7f2578040460 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f258c125c10 tx=0x7f2584008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:11.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.845+0000 7f25737fe640 1 -- 172.21.15.100:0/1219680756 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f257c030d50 con 0x7f258c138240 2024-09-13T11:53:11.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.845+0000 7f25737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:11.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:11.845+0000 7f25737fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:12.018 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.016+0000 7f25912b6640 1 -- 172.21.15.100:0/1219680756 --> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f258c1cd750 con 0x7f257803dfa0 2024-09-13T11:53:12.019 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.017+0000 7f25737fe640 1 -- 172.21.15.100:0/1219680756 <== mgr.4114 v2:172.21.15.122:6824/441789830 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34178 (secure 0 0 0) 0x7f258c1cd750 con 0x7f257803dfa0 2024-09-13T11:53:12.020 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:53:12.020 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-13T11:53:12.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.019+0000 7f25912b6640 1 -- 172.21.15.100:0/1219680756 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f257803dfa0 msgr2=0x7f2578040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.019+0000 7f25912b6640 1 --2- 172.21.15.100:0/1219680756 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f257803dfa0 0x7f2578040460 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f258c125c10 tx=0x7f2584008040 comp rx=0 tx=0).stop 2024-09-13T11:53:12.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.019+0000 7f25912b6640 10 monclient: shutdown 2024-09-13T11:53:12.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.019+0000 7f25912b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:12.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.019+0000 7f25912b6640 1 -- 172.21.15.100:0/1219680756 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f258c138240 msgr2=0x7f258c136f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.019+0000 7f25912b6640 1 --2- 172.21.15.100:0/1219680756 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f258c138240 0x7f258c136f00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f257c031000 tx=0x7f257c0046c0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.019+0000 7f25912b6640 1 -- 172.21.15.100:0/1219680756 shutdown_connections 2024-09-13T11:53:12.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.019+0000 7f25912b6640 1 --2- 172.21.15.100:0/1219680756 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f257803dfa0 0x7f2578040460 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.019+0000 7f25912b6640 1 --2- 172.21.15.100:0/1219680756 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f258c138240 0x7f258c136f00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.019+0000 7f25912b6640 1 --2- 172.21.15.100:0/1219680756 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f258c137e40 0x7f258c1369c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.019+0000 7f25912b6640 1 --2- 172.21.15.100:0/1219680756 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f258c137a60 0x7f258c1cca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.019+0000 7f25912b6640 1 -- 172.21.15.100:0/1219680756 >> 172.21.15.100:0/1219680756 conn(0x7f258c0aa580 msgr2=0x7f258c0a4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:12.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.019+0000 7f25912b6640 1 -- 172.21.15.100:0/1219680756 shutdown_connections 2024-09-13T11:53:12.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.019+0000 7f25912b6640 1 -- 172.21.15.100:0/1219680756 wait complete. 2024-09-13T11:53:12.032 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-13T11:53:11.212494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163332,"kb_used_data":1136,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067708,"statfs":{"total":579820584960,"available":579653332992,"internally_reserved":0,"allocated":1163264,"data_stored":1397488,"data_compressed":59246,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001321"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.266756+0000","last_change":"2024-09-13T11:53:06.266756+0000","last_active":"2024-09-13T11:53:06.266756+0000","last_peered":"2024-09-13T11:53:06.266756+0000","last_clean":"2024-09-13T11:53:06.266756+0000","last_became_active":"2024-09-13T11:53:04.566167+0000","last_became_peered":"2024-09-13T11:53:04.566167+0000","last_unstale":"2024-09-13T11:53:06.266756+0000","last_undegraded":"2024-09-13T11:53:06.266756+0000","last_fullsized":"2024-09-13T11:53:06.266756+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:41.983816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084363899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.266077+0000","last_change":"2024-09-13T11:53:06.266346+0000","last_active":"2024-09-13T11:53:06.266077+0000","last_peered":"2024-09-13T11:53:06.266077+0000","last_clean":"2024-09-13T11:53:06.266077+0000","last_became_active":"2024-09-13T11:53:04.258975+0000","last_became_peered":"2024-09-13T11:53:04.258975+0000","last_unstale":"2024-09-13T11:53:06.266077+0000","last_undegraded":"2024-09-13T11:53:06.266077+0000","last_fullsized":"2024-09-13T11:53:06.266077+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:36.773393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055464299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.266446+0000","last_change":"2024-09-13T11:53:06.266446+0000","last_active":"2024-09-13T11:53:06.266446+0000","last_peered":"2024-09-13T11:53:06.266446+0000","last_clean":"2024-09-13T11:53:06.266446+0000","last_became_active":"2024-09-13T11:53:04.262475+0000","last_became_peered":"2024-09-13T11:53:04.262475+0000","last_unstale":"2024-09-13T11:53:06.266446+0000","last_undegraded":"2024-09-13T11:53:06.266446+0000","last_fullsized":"2024-09-13T11:53:06.266446+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:24:12.639511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055213099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.266124+0000","last_change":"2024-09-13T11:53:06.266390+0000","last_active":"2024-09-13T11:53:06.266124+0000","last_peered":"2024-09-13T11:53:06.266124+0000","last_clean":"2024-09-13T11:53:06.266124+0000","last_became_active":"2024-09-13T11:53:04.262392+0000","last_became_peered":"2024-09-13T11:53:04.262392+0000","last_unstale":"2024-09-13T11:53:06.266124+0000","last_undegraded":"2024-09-13T11:53:06.266124+0000","last_fullsized":"2024-09-13T11:53:06.266124+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:45.666214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057483700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.272635+0000","last_change":"2024-09-13T11:53:06.272635+0000","last_active":"2024-09-13T11:53:06.272635+0000","last_peered":"2024-09-13T11:53:06.272635+0000","last_clean":"2024-09-13T11:53:06.272635+0000","last_became_active":"2024-09-13T11:53:04.264659+0000","last_became_peered":"2024-09-13T11:53:04.264659+0000","last_unstale":"2024-09-13T11:53:06.272635+0000","last_undegraded":"2024-09-13T11:53:06.272635+0000","last_fullsized":"2024-09-13T11:53:06.272635+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:53:49.774126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010464509999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.265979+0000","last_change":"2024-09-13T11:53:06.266298+0000","last_active":"2024-09-13T11:53:06.265979+0000","last_peered":"2024-09-13T11:53:06.265979+0000","last_clean":"2024-09-13T11:53:06.265979+0000","last_became_active":"2024-09-13T11:53:04.270146+0000","last_became_peered":"2024-09-13T11:53:04.270146+0000","last_unstale":"2024-09-13T11:53:06.265979+0000","last_undegraded":"2024-09-13T11:53:06.265979+0000","last_fullsized":"2024-09-13T11:53:06.265979+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:38.439658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061806000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.266345+0000","last_change":"2024-09-13T11:53:06.266345+0000","last_active":"2024-09-13T11:53:06.266345+0000","last_peered":"2024-09-13T11:53:06.266345+0000","last_clean":"2024-09-13T11:53:06.266345+0000","last_became_active":"2024-09-13T11:53:04.262197+0000","last_became_peered":"2024-09-13T11:53:04.262197+0000","last_unstale":"2024-09-13T11:53:06.266345+0000","last_undegraded":"2024-09-13T11:53:06.266345+0000","last_fullsized":"2024-09-13T11:53:06.266345+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:05:51.189944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044149899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":21,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.271834+0000","last_change":"2024-09-13T11:53:06.272071+0000","last_active":"2024-09-13T11:53:06.271834+0000","last_peered":"2024-09-13T11:53:06.271834+0000","last_clean":"2024-09-13T11:53:06.271834+0000","last_became_active":"2024-09-13T11:53:04.270138+0000","last_became_peered":"2024-09-13T11:53:04.270138+0000","last_unstale":"2024-09-13T11:53:06.271834+0000","last_undegraded":"2024-09-13T11:53:06.271834+0000","last_fullsized":"2024-09-13T11:53:06.271834+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:17.090753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047405,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":111,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.265801+0000","last_change":"2024-09-13T11:52:57.706946+0000","last_active":"2024-09-13T11:53:06.265801+0000","last_peered":"2024-09-13T11:53:06.265801+0000","last_clean":"2024-09-13T11:53:06.265801+0000","last_became_active":"2024-09-13T11:52:57.705894+0000","last_became_peered":"2024-09-13T11:52:57.705894+0000","last_unstale":"2024-09-13T11:53:06.265801+0000","last_undegraded":"2024-09-13T11:53:06.265801+0000","last_fullsized":"2024-09-13T11:53:06.265801+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:52:56.225375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:52:56.225375+0000","last_clean_scrub_stamp":"2024-09-13T11:52:56.225375+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:05.299261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59246,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":172032,"data_stored":56599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607557,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":233472,"data_stored":585527,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607558,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":585527,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":176128,"data_stored":56618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":172032,"data_stored":56599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640262,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":176128,"data_stored":56618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:53:12.033 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:53:12.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.127+0000 7ff09cdc2640 10 monclient: get_monmap_and_config 2024-09-13T11:53:12.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.127+0000 7ff09cdc2640 10 monclient: build_initial_monmap 2024-09-13T11:53:12.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.127+0000 7ff09cdc2640 10 monclient: monmap: 2024-09-13T11:53:12.130 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:12.130 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:12.130 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:12.129401+0000 2024-09-13T11:53:12.130 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:12.129401+0000 2024-09-13T11:53:12.130 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:12.131 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.131 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:12.131 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:12.131 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:12.131 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:12.131 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:12.131 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.128+0000 7ff09cdc2640 1 Processor -- start 2024-09-13T11:53:12.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.128+0000 7ff09cdc2640 1 -- start start 2024-09-13T11:53:12.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.128+0000 7ff09cdc2640 10 monclient: init 2024-09-13T11:53:12.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.128+0000 7ff09cdc2640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:12.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.128+0000 7ff09cdc2640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-13T11:53:12.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.128+0000 7ff09cdc2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff0980995a0 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:12.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.128+0000 7ff09cdc2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff09809b740 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:12.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.128+0000 7ff09cdc2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff09813de50 0x7ff098140240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.128+0000 7ff09cdc2640 10 monclient(hunting): picked mon.noname-c con 0x7ff09813de50 addr v2:172.21.15.122:3300/0 2024-09-13T11:53:12.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.128+0000 7ff09cdc2640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff09809bb50 con 0x7ff0980995a0 2024-09-13T11:53:12.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.128+0000 7ff09cdc2640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff09806d9a0 con 0x7ff09809b740 2024-09-13T11:53:12.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.128+0000 7ff09cdc2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.128+0000 7ff09cdc2640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7ff09809d350 con 0x7ff09813de50 2024-09-13T11:53:12.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.128+0000 7ff09cdc2640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:12.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.128+0000 7ff09cdc2640 10 monclient(hunting): authenticate will time out at 772.722788s 2024-09-13T11:53:12.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.128+0000 7ff09659c640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7ff0980995a0 0x7ff098099980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:58052/0 (socket says 172.21.15.100:58052) 2024-09-13T11:53:12.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.128+0000 7ff09659c640 1 -- 172.21.15.100:0/2927180245 learned_addr learned my addr 172.21.15.100:0/2927180245 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:53:12.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 1 -- 172.21.15.100:0/2927180245 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2260593889 0 0) 0x7ff09809bb50 con 0x7ff0980995a0 2024-09-13T11:53:12.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 10 monclient(hunting): _negotiate 2024-09-13T11:53:12.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:12.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:12.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 10 monclient(hunting): my global_id is 4404 2024-09-13T11:53:12.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 1 -- 172.21.15.100:0/2927180245 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff080003520 con 0x7ff0980995a0 2024-09-13T11:53:12.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff096d9d640 1 --2- 172.21.15.100:0/2927180245 >> v2:172.21.15.122:3300/0 conn(0x7ff09813de50 0x7ff098140240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff096d9d640 10 monclient(hunting): get_auth_request con 0x7ff09813de50 auth_method 0 2024-09-13T11:53:12.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff096d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:12.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff096d9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:12.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff096d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:12.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 1 -- 172.21.15.100:0/2927180245 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2069084038 0 0) 0x7ff080003520 con 0x7ff0980995a0 2024-09-13T11:53:12.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 1 -- 172.21.15.100:0/2927180245 >> v2:172.21.15.122:3300/0 conn(0x7ff09813de50 msgr2=0x7ff098140240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:12.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 1 --2- 172.21.15.100:0/2927180245 >> v2:172.21.15.122:3300/0 conn(0x7ff09813de50 0x7ff098140240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 1 -- 172.21.15.100:0/2927180245 >> v1:172.21.15.122:6789/0 conn(0x7ff09809b740 legacy=0x7ff09813c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:12.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 10 monclient: _finish_hunting 0 2024-09-13T11:53:12.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 1 monclient: found mon.noname-a-legacy 2024-09-13T11:53:12.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:12.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.100:6789/0 2024-09-13T11:53:12.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 1 -- 172.21.15.100:0/2927180245 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff09809a4a0 con 0x7ff0980995a0 2024-09-13T11:53:12.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 10 monclient: _finish_auth 0 2024-09-13T11:53:12.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 10 monclient: _check_auth_tickets 2024-09-13T11:53:12.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:12.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 1 -- 172.21.15.100:0/2927180245 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 626700363 0 0) 0x7ff0880016f0 con 0x7ff0980995a0 2024-09-13T11:53:12.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09cdc2640 5 monclient: authenticate success, global_id 4404 2024-09-13T11:53:12.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09cdc2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:12.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:12.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff096d9d640 1 --2- 172.21.15.100:0/2927180245 >> v2:172.21.15.122:3300/0 conn(0x7ff09813de50 0x7ff098140240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:53:12.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-13T11:53:12.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 10 monclient: dump: 2024-09-13T11:53:12.145 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:12.145 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:12.145 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.145 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.145 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:12.145 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.145 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:12.145 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:12.145 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:12.145 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.145 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] but i'm connected to v1:172.21.15.100:6789/0, reconnecting 2024-09-13T11:53:12.145 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:12.145 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 1 -- 172.21.15.100:0/2927180245 >> v1:172.21.15.100:6789/0 conn(0x7ff0980995a0 legacy=0x7ff098099980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.146 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:53:12.146 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 1 --2- 172.21.15.100:0/2927180245 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff080003a50 0x7ff080003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.146 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 10 monclient(hunting): picked mon.a con 0x7ff080003a50 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:12.146 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 1 --2- 172.21.15.100:0/2927180245 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff080004800 0x7ff080004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.146 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 10 monclient(hunting): picked mon.c con 0x7ff080004800 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:12.146 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 1 --2- 172.21.15.100:0/2927180245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0800053a0 0x7ff08000ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.146 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 10 monclient(hunting): picked mon.b con 0x7ff0800053a0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:12.146 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.146 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 1 -- 172.21.15.100:0/2927180245 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7ff08000c2b0 con 0x7ff080003a50 2024-09-13T11:53:12.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 1 -- 172.21.15.100:0/2927180245 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff08000c460 con 0x7ff0800053a0 2024-09-13T11:53:12.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 1 -- 172.21.15.100:0/2927180245 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7ff08000c660 con 0x7ff080004800 2024-09-13T11:53:12.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:12.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09559a640 10 monclient(hunting): _finish_auth 0 2024-09-13T11:53:12.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09cdc2640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:12.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09659c640 1 --2- 172.21.15.100:0/2927180245 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff080003a50 0x7ff080003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09659c640 10 monclient(hunting): get_auth_request con 0x7ff080003a50 auth_method 0 2024-09-13T11:53:12.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:12.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09659c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:12.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:53:12.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff095d9b640 1 --2- 172.21.15.100:0/2927180245 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff080004800 0x7ff080004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff095d9b640 10 monclient(hunting): get_auth_request con 0x7ff080004800 auth_method 0 2024-09-13T11:53:12.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff095d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:12.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff095d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:12.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff095d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:53:12.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:12.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:12.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff096d9d640 1 --2- 172.21.15.100:0/2927180245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0800053a0 0x7ff08000ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:53:12.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff096d9d640 10 monclient(hunting): get_auth_request con 0x7ff0800053a0 auth_method 0 2024-09-13T11:53:12.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff096d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:12.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff096d9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:12.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff096d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:53:12.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff095d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:12.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff095d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:12.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff095d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:53:12.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff096d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:12.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff096d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:12.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff096d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:53:12.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09659c640 10 monclient(hunting): handle_auth_done global_id 4404 payload 293 2024-09-13T11:53:12.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09659c640 1 -- 172.21.15.100:0/2927180245 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff080004800 msgr2=0x7ff080004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:12.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09659c640 1 --2- 172.21.15.100:0/2927180245 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff080004800 0x7ff080004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09659c640 1 -- 172.21.15.100:0/2927180245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0800053a0 msgr2=0x7ff08000ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:12.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09659c640 1 --2- 172.21.15.100:0/2927180245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0800053a0 0x7ff08000ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09659c640 10 monclient: _finish_hunting 0 2024-09-13T11:53:12.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09659c640 1 monclient: found mon.a 2024-09-13T11:53:12.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff096d9d640 1 -- 172.21.15.100:0/2927180245 reap_dead start 2024-09-13T11:53:12.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:12.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09659c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:12.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09659c640 1 -- 172.21.15.100:0/2927180245 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff080003520 con 0x7ff080003a50 2024-09-13T11:53:12.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09659c640 10 monclient: _finish_auth 0 2024-09-13T11:53:12.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09659c640 10 monclient: _check_auth_tickets 2024-09-13T11:53:12.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.129+0000 7ff09659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:12.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.130+0000 7ff09659c640 1 --2- 172.21.15.100:0/2927180245 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff080003a50 0x7ff080003e70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff08800c290 tx=0x7ff080005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b18676111f4b9733 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:12.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.130+0000 7ff09559a640 1 -- 172.21.15.100:0/2927180245 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff088007d50 con 0x7ff080003a50 2024-09-13T11:53:12.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.130+0000 7ff09559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:12.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.130+0000 7ff09559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:12.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.130+0000 7ff09559a640 10 monclient: dump: 2024-09-13T11:53:12.150 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:12.150 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:12.151 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.151 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.151 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:12.151 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.151 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:12.151 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:12.151 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:12.151 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.130+0000 7ff09559a640 1 -- 172.21.15.100:0/2927180245 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff08802e070 con 0x7ff080003a50 2024-09-13T11:53:12.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.130+0000 7ff09cdc2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:12.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.130+0000 7ff09559a640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:12.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.130+0000 7ff09559a640 1 -- 172.21.15.100:0/2927180245 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff088005d70 con 0x7ff080003a50 2024-09-13T11:53:12.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.130+0000 7ff09cdc2640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:12.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.130+0000 7ff09cdc2640 4 set_mon_vals no callback set 2024-09-13T11:53:12.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.130+0000 7ff09559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:12.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.130+0000 7ff09559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:12.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.130+0000 7ff09559a640 10 monclient: dump: 2024-09-13T11:53:12.152 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:12.152 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:12.152 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.152 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.152 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:12.152 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.152 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:12.152 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:12.152 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:12.153 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.131+0000 7ff09cdc2640 10 monclient: shutdown 2024-09-13T11:53:12.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.131+0000 7ff09cdc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:12.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.131+0000 7ff09cdc2640 1 -- 172.21.15.100:0/2927180245 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff080003a50 msgr2=0x7ff080003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.131+0000 7ff09cdc2640 1 --2- 172.21.15.100:0/2927180245 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff080003a50 0x7ff080003e70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff08800c290 tx=0x7ff080005830 comp rx=0 tx=0).stop 2024-09-13T11:53:12.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.131+0000 7ff09cdc2640 1 -- 172.21.15.100:0/2927180245 shutdown_connections 2024-09-13T11:53:12.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.131+0000 7ff09cdc2640 1 --2- 172.21.15.100:0/2927180245 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff080003a50 0x7ff080003e70 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.131+0000 7ff09cdc2640 1 -- 172.21.15.100:0/2927180245 >> 172.21.15.100:0/2927180245 conn(0x7ff0980a2e70 msgr2=0x7ff0980a3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:12.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.131+0000 7ff09cdc2640 1 -- 172.21.15.100:0/2927180245 shutdown_connections 2024-09-13T11:53:12.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.131+0000 7ff09cdc2640 1 -- 172.21.15.100:0/2927180245 wait complete. 2024-09-13T11:53:12.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09cdc2640 10 monclient: build_initial_monmap 2024-09-13T11:53:12.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09cdc2640 10 monclient: monmap: 2024-09-13T11:53:12.153 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:12.154 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:12.154 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:12.154 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:12.154 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:12.154 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.154 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:12.154 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:12.154 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:12.154 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09cdc2640 1 Processor -- start 2024-09-13T11:53:12.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09cdc2640 1 -- start start 2024-09-13T11:53:12.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09cdc2640 10 monclient: init 2024-09-13T11:53:12.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09cdc2640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:12.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09cdc2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:53:12.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09cdc2640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff080003a50 0x7ff098096170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09cdc2640 10 monclient(hunting): picked mon.noname-a con 0x7ff080003a50 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:12.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09cdc2640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff098093720 0x7ff0980909d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09cdc2640 10 monclient(hunting): picked mon.noname-c con 0x7ff098093720 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:12.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09cdc2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff098090f10 0x7ff0980913a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09cdc2640 10 monclient(hunting): picked mon.noname-b con 0x7ff098090f10 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:12.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09cdc2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09cdc2640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7ff09806d5f0 con 0x7ff080003a50 2024-09-13T11:53:12.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09cdc2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09cdc2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff09806d470 con 0x7ff098090f10 2024-09-13T11:53:12.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09cdc2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09cdc2640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7ff098099980 con 0x7ff098093720 2024-09-13T11:53:12.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09cdc2640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:12.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09cdc2640 10 monclient(hunting): authenticate will time out at 772.726897s 2024-09-13T11:53:12.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09659c640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff080003a50 0x7ff098096170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09659c640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff080003a50 0x7ff098096170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:49818/0 (socket says 172.21.15.100:49818) 2024-09-13T11:53:12.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09659c640 1 -- 172.21.15.100:0/4035863374 learned_addr learned my addr 172.21.15.100:0/4035863374 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:12.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09659c640 10 monclient(hunting): get_auth_request con 0x7ff080003a50 auth_method 0 2024-09-13T11:53:12.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:12.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09659c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:12.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff09659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:12.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.132+0000 7ff095d9b640 1 --2- 172.21.15.100:0/4035863374 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff098093720 0x7ff0980909d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff095d9b640 10 monclient(hunting): get_auth_request con 0x7ff098093720 auth_method 0 2024-09-13T11:53:12.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff095d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:12.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff095d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:12.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff095d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:12.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff09659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:12.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff09659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:12.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff09659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:12.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff095d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:12.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff095d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:12.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff096d9d640 1 --2- 172.21.15.100:0/4035863374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff098090f10 0x7ff0980913a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff095d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:12.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff096d9d640 10 monclient(hunting): get_auth_request con 0x7ff098090f10 auth_method 0 2024-09-13T11:53:12.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff096d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:12.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff096d9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:12.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff096d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:12.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff096d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:12.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff096d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:12.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff096d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:12.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff09659c640 10 monclient(hunting): handle_auth_done global_id 4407 payload 824 2024-09-13T11:53:12.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff09659c640 1 -- 172.21.15.100:0/4035863374 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff098093720 msgr2=0x7ff0980909d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff09659c640 1 --2- 172.21.15.100:0/4035863374 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff098093720 0x7ff0980909d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff09659c640 1 -- 172.21.15.100:0/4035863374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff098090f10 msgr2=0x7ff0980913a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff09659c640 1 --2- 172.21.15.100:0/4035863374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff098090f10 0x7ff0980913a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.159 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff09659c640 10 monclient: _finish_hunting 0 2024-09-13T11:53:12.159 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff09659c640 1 monclient: found mon.noname-a 2024-09-13T11:53:12.159 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff09659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:12.159 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff09659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:12.159 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff09659c640 1 -- 172.21.15.100:0/4035863374 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff09813da80 con 0x7ff080003a50 2024-09-13T11:53:12.159 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff095d9b640 1 --2- 172.21.15.100:0/4035863374 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff098093720 0x7ff0980909d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:12.159 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff09659c640 1 --2- 172.21.15.100:0/4035863374 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff080003a50 0x7ff098096170 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff08802ad70 tx=0x7ff09809a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:12.159 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff086ffd640 1 -- 172.21.15.100:0/4035863374 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff088043070 con 0x7ff080003a50 2024-09-13T11:53:12.159 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff086ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:12.159 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff086ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:12.159 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff086ffd640 10 monclient: dump: 2024-09-13T11:53:12.159 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:12.160 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:12.160 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.160 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.160 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:12.160 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.160 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:12.160 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:12.160 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:12.160 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.160 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff086ffd640 10 monclient: _finish_auth 0 2024-09-13T11:53:12.160 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff086ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:53:12.160 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff086ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:12.161 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff086ffd640 1 -- 172.21.15.100:0/4035863374 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff088007480 con 0x7ff080003a50 2024-09-13T11:53:12.161 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff09cdc2640 5 monclient: authenticate success, global_id 4407 2024-09-13T11:53:12.161 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff086ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:12.161 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff086ffd640 1 -- 172.21.15.100:0/4035863374 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff088007d50 con 0x7ff080003a50 2024-09-13T11:53:12.161 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff09cdc2640 10 monclient: _renew_subs 2024-09-13T11:53:12.161 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff09cdc2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:12.161 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff09cdc2640 1 -- 172.21.15.100:0/4035863374 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0980947f0 con 0x7ff080003a50 2024-09-13T11:53:12.162 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff086ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:12.162 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff086ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:12.162 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff094d99640 4 set_mon_vals no callback set 2024-09-13T11:53:12.162 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff086ffd640 10 monclient: dump: 2024-09-13T11:53:12.162 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:12.162 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:12.162 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.162 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.162 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:12.162 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.162 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:12.162 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:12.162 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:12.162 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff09cdc2640 10 monclient: _renew_subs 2024-09-13T11:53:12.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff09cdc2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:12.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.133+0000 7ff09cdc2640 1 -- 172.21.15.100:0/4035863374 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0980a1400 con 0x7ff080003a50 2024-09-13T11:53:12.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.134+0000 7ff086ffd640 1 -- 172.21.15.100:0/4035863374 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7ff088033470 con 0x7ff080003a50 2024-09-13T11:53:12.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.134+0000 7ff09cdc2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:12.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.134+0000 7ff09cdc2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:12.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.134+0000 7ff09cdc2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:12.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.134+0000 7ff09cdc2640 1 -- 172.21.15.100:0/4035863374 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff098099d10 con 0x7ff080003a50 2024-09-13T11:53:12.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.134+0000 7ff086ffd640 1 --2- 172.21.15.100:0/4035863374 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff07803dfd0 0x7ff078040490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.134+0000 7ff086ffd640 1 -- 172.21.15.100:0/4035863374 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4374+0+0 (secure 0 0 0) 0x7ff0880720d0 con 0x7ff080003a50 2024-09-13T11:53:12.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.134+0000 7ff095d9b640 1 --2- 172.21.15.100:0/4035863374 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff07803dfd0 0x7ff078040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.134+0000 7ff095d9b640 10 monclient: get_auth_request con 0x7ff07803dfd0 auth_method 0 2024-09-13T11:53:12.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.135+0000 7ff095d9b640 1 --2- 172.21.15.100:0/4035863374 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff07803dfd0 0x7ff078040490 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff098093b00 tx=0x7ff08c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:12.164 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.139+0000 7ff086ffd640 1 -- 172.21.15.100:0/4035863374 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7ff08803cd50 con 0x7ff080003a50 2024-09-13T11:53:12.164 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.139+0000 7ff086ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:12.164 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.139+0000 7ff086ffd640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:12.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.312+0000 7ff09cdc2640 1 -- 172.21.15.100:0/4035863374 --> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff098092ff0 con 0x7ff07803dfd0 2024-09-13T11:53:12.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.314+0000 7ff086ffd640 1 -- 172.21.15.100:0/4035863374 <== mgr.4114 v2:172.21.15.122:6824/441789830 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34178 (secure 0 0 0) 0x7ff098092ff0 con 0x7ff07803dfd0 2024-09-13T11:53:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:53:12.317 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-13T11:53:12.318 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.316+0000 7ff09cdc2640 1 -- 172.21.15.100:0/4035863374 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff07803dfd0 msgr2=0x7ff078040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.316+0000 7ff09cdc2640 1 --2- 172.21.15.100:0/4035863374 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff07803dfd0 0x7ff078040490 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff098093b00 tx=0x7ff08c009040 comp rx=0 tx=0).stop 2024-09-13T11:53:12.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.316+0000 7ff09cdc2640 10 monclient: shutdown 2024-09-13T11:53:12.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.316+0000 7ff09cdc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:12.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.316+0000 7ff09cdc2640 1 -- 172.21.15.100:0/4035863374 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff080003a50 msgr2=0x7ff098096170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.316+0000 7ff09cdc2640 1 --2- 172.21.15.100:0/4035863374 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff080003a50 0x7ff098096170 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff08802ad70 tx=0x7ff09809a180 comp rx=0 tx=0).stop 2024-09-13T11:53:12.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.316+0000 7ff09cdc2640 1 -- 172.21.15.100:0/4035863374 shutdown_connections 2024-09-13T11:53:12.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.316+0000 7ff09cdc2640 1 --2- 172.21.15.100:0/4035863374 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff07803dfd0 0x7ff078040490 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.316+0000 7ff09cdc2640 1 --2- 172.21.15.100:0/4035863374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff098090f10 0x7ff0980913a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.316+0000 7ff09cdc2640 1 --2- 172.21.15.100:0/4035863374 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff098093720 0x7ff0980909d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.316+0000 7ff09cdc2640 1 --2- 172.21.15.100:0/4035863374 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff080003a50 0x7ff098096170 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.316+0000 7ff09cdc2640 1 -- 172.21.15.100:0/4035863374 >> 172.21.15.100:0/4035863374 conn(0x7ff0980a2e70 msgr2=0x7ff0980a4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:12.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.316+0000 7ff09cdc2640 1 -- 172.21.15.100:0/4035863374 shutdown_connections 2024-09-13T11:53:12.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.316+0000 7ff09cdc2640 1 -- 172.21.15.100:0/4035863374 wait complete. 2024-09-13T11:53:12.329 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-13T11:53:11.212494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163332,"kb_used_data":1136,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067708,"statfs":{"total":579820584960,"available":579653332992,"internally_reserved":0,"allocated":1163264,"data_stored":1397488,"data_compressed":59246,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001321"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.266756+0000","last_change":"2024-09-13T11:53:06.266756+0000","last_active":"2024-09-13T11:53:06.266756+0000","last_peered":"2024-09-13T11:53:06.266756+0000","last_clean":"2024-09-13T11:53:06.266756+0000","last_became_active":"2024-09-13T11:53:04.566167+0000","last_became_peered":"2024-09-13T11:53:04.566167+0000","last_unstale":"2024-09-13T11:53:06.266756+0000","last_undegraded":"2024-09-13T11:53:06.266756+0000","last_fullsized":"2024-09-13T11:53:06.266756+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:41.983816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084363899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.266077+0000","last_change":"2024-09-13T11:53:06.266346+0000","last_active":"2024-09-13T11:53:06.266077+0000","last_peered":"2024-09-13T11:53:06.266077+0000","last_clean":"2024-09-13T11:53:06.266077+0000","last_became_active":"2024-09-13T11:53:04.258975+0000","last_became_peered":"2024-09-13T11:53:04.258975+0000","last_unstale":"2024-09-13T11:53:06.266077+0000","last_undegraded":"2024-09-13T11:53:06.266077+0000","last_fullsized":"2024-09-13T11:53:06.266077+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:36.773393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055464299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.266446+0000","last_change":"2024-09-13T11:53:06.266446+0000","last_active":"2024-09-13T11:53:06.266446+0000","last_peered":"2024-09-13T11:53:06.266446+0000","last_clean":"2024-09-13T11:53:06.266446+0000","last_became_active":"2024-09-13T11:53:04.262475+0000","last_became_peered":"2024-09-13T11:53:04.262475+0000","last_unstale":"2024-09-13T11:53:06.266446+0000","last_undegraded":"2024-09-13T11:53:06.266446+0000","last_fullsized":"2024-09-13T11:53:06.266446+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:24:12.639511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055213099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.266124+0000","last_change":"2024-09-13T11:53:06.266390+0000","last_active":"2024-09-13T11:53:06.266124+0000","last_peered":"2024-09-13T11:53:06.266124+0000","last_clean":"2024-09-13T11:53:06.266124+0000","last_became_active":"2024-09-13T11:53:04.262392+0000","last_became_peered":"2024-09-13T11:53:04.262392+0000","last_unstale":"2024-09-13T11:53:06.266124+0000","last_undegraded":"2024-09-13T11:53:06.266124+0000","last_fullsized":"2024-09-13T11:53:06.266124+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:45.666214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057483700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.272635+0000","last_change":"2024-09-13T11:53:06.272635+0000","last_active":"2024-09-13T11:53:06.272635+0000","last_peered":"2024-09-13T11:53:06.272635+0000","last_clean":"2024-09-13T11:53:06.272635+0000","last_became_active":"2024-09-13T11:53:04.264659+0000","last_became_peered":"2024-09-13T11:53:04.264659+0000","last_unstale":"2024-09-13T11:53:06.272635+0000","last_undegraded":"2024-09-13T11:53:06.272635+0000","last_fullsized":"2024-09-13T11:53:06.272635+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:53:49.774126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010464509999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.265979+0000","last_change":"2024-09-13T11:53:06.266298+0000","last_active":"2024-09-13T11:53:06.265979+0000","last_peered":"2024-09-13T11:53:06.265979+0000","last_clean":"2024-09-13T11:53:06.265979+0000","last_became_active":"2024-09-13T11:53:04.270146+0000","last_became_peered":"2024-09-13T11:53:04.270146+0000","last_unstale":"2024-09-13T11:53:06.265979+0000","last_undegraded":"2024-09-13T11:53:06.265979+0000","last_fullsized":"2024-09-13T11:53:06.265979+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:38.439658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061806000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.266345+0000","last_change":"2024-09-13T11:53:06.266345+0000","last_active":"2024-09-13T11:53:06.266345+0000","last_peered":"2024-09-13T11:53:06.266345+0000","last_clean":"2024-09-13T11:53:06.266345+0000","last_became_active":"2024-09-13T11:53:04.262197+0000","last_became_peered":"2024-09-13T11:53:04.262197+0000","last_unstale":"2024-09-13T11:53:06.266345+0000","last_undegraded":"2024-09-13T11:53:06.266345+0000","last_fullsized":"2024-09-13T11:53:06.266345+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:05:51.189944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044149899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":21,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.271834+0000","last_change":"2024-09-13T11:53:06.272071+0000","last_active":"2024-09-13T11:53:06.271834+0000","last_peered":"2024-09-13T11:53:06.271834+0000","last_clean":"2024-09-13T11:53:06.271834+0000","last_became_active":"2024-09-13T11:53:04.270138+0000","last_became_peered":"2024-09-13T11:53:04.270138+0000","last_unstale":"2024-09-13T11:53:06.271834+0000","last_undegraded":"2024-09-13T11:53:06.271834+0000","last_fullsized":"2024-09-13T11:53:06.271834+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:17.090753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047405,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":111,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T11:53:06.265801+0000","last_change":"2024-09-13T11:52:57.706946+0000","last_active":"2024-09-13T11:53:06.265801+0000","last_peered":"2024-09-13T11:53:06.265801+0000","last_clean":"2024-09-13T11:53:06.265801+0000","last_became_active":"2024-09-13T11:52:57.705894+0000","last_became_peered":"2024-09-13T11:52:57.705894+0000","last_unstale":"2024-09-13T11:53:06.265801+0000","last_undegraded":"2024-09-13T11:53:06.265801+0000","last_fullsized":"2024-09-13T11:53:06.265801+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:52:56.225375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:52:56.225375+0000","last_clean_scrub_stamp":"2024-09-13T11:52:56.225375+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:05.299261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59246,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":172032,"data_stored":56599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607557,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":233472,"data_stored":585527,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607558,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":585527,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":176128,"data_stored":56618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":172032,"data_stored":56599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640262,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":176128,"data_stored":56618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:53:12.330 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-13T11:53:12.330 DEBUG:tasks.mon_thrash:Yielding 2024-09-13T11:53:12.330 INFO:teuthology.run_tasks:Running task exec... 2024-09-13T11:53:12.341 INFO:teuthology.task.exec:Executing custom commands... 2024-09-13T11:53:12.341 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T11:53:12.341 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:53:12.341 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-13T11:53:12.341 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726228392, revive delay: 90.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: True, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-13T11:53:12.342 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-13T11:53:12.342 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:53:12.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.437+0000 7f48f291e640 10 monclient: get_monmap_and_config 2024-09-13T11:53:12.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.437+0000 7f48f291e640 10 monclient: build_initial_monmap 2024-09-13T11:53:12.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.437+0000 7f48f291e640 10 monclient: monmap: 2024-09-13T11:53:12.440 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:12.440 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:12.440 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:12.439085+0000 2024-09-13T11:53:12.440 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:12.439085+0000 2024-09-13T11:53:12.440 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:12.440 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.440 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:12.440 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:12.440 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:12.440 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:12.440 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:12.440 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.448 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.437+0000 7f48f291e640 1 Processor -- start 2024-09-13T11:53:12.448 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48f291e640 1 -- start start 2024-09-13T11:53:12.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48f291e640 10 monclient: init 2024-09-13T11:53:12.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48f291e640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:12.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48f291e640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-13T11:53:12.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48f291e640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f48ec137a60 0x7f48ec0a2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48f291e640 10 monclient(hunting): picked mon.noname-a con 0x7f48ec137a60 addr v2:172.21.15.100:3300/0 2024-09-13T11:53:12.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48f291e640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48ec137e40 0x7f48ec0a4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48f291e640 10 monclient(hunting): picked mon.noname-b con 0x7f48ec137e40 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:12.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48f291e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f48ec138270 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:12.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48f291e640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48ec0a5be0 con 0x7f48ec138270 2024-09-13T11:53:12.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48f291e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48f291e640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f48ec0a4410 con 0x7f48ec137a60 2024-09-13T11:53:12.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48f291e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48f291e640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f48ec0a76b0 con 0x7f48ec137e40 2024-09-13T11:53:12.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48f291e640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:12.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48f291e640 10 monclient(hunting): authenticate will time out at 773.032507s 2024-09-13T11:53:12.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48eb7fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48ec137e40 0x7f48ec0a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48eb7fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48ec137e40 0x7f48ec0a4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:35168/0 (socket says 172.21.15.100:35168) 2024-09-13T11:53:12.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48eb7fe640 1 -- 172.21.15.100:0/967345572 learned_addr learned my addr 172.21.15.100:0/967345572 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:12.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48eb7fe640 10 monclient(hunting): get_auth_request con 0x7f48ec137e40 auth_method 0 2024-09-13T11:53:12.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:12.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:12.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:12.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48ebfff640 1 --2- 172.21.15.100:0/967345572 >> v2:172.21.15.100:3300/0 conn(0x7f48ec137a60 0x7f48ec0a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:12.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:12.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:12.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48ebfff640 10 monclient(hunting): get_auth_request con 0x7f48ec137a60 auth_method 0 2024-09-13T11:53:12.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:12.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:12.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:12.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:12.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:12.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:12.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48eb7fe640 10 monclient(hunting): handle_auth_done global_id 4382 payload 274 2024-09-13T11:53:12.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48eb7fe640 1 -- 172.21.15.100:0/967345572 >> v2:172.21.15.100:3300/0 conn(0x7f48ec137a60 msgr2=0x7f48ec0a2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:12.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48eb7fe640 1 --2- 172.21.15.100:0/967345572 >> v2:172.21.15.100:3300/0 conn(0x7f48ec137a60 0x7f48ec0a2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48eb7fe640 1 -- 172.21.15.100:0/967345572 >> v1:172.21.15.100:6789/0 conn(0x7f48ec138270 legacy=0x7f48ec0a4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48eb7fe640 10 monclient: _finish_hunting 0 2024-09-13T11:53:12.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48eb7fe640 1 monclient: found mon.noname-b 2024-09-13T11:53:12.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:12.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48eb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:53:12.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48eb7fe640 1 -- 172.21.15.100:0/967345572 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48ec0a19e0 con 0x7f48ec137e40 2024-09-13T11:53:12.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.438+0000 7f48eaffd640 1 -- 172.21.15.100:0/967345572 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1243009429 0 0) 0x7f48ec0a5be0 con 0x7f48ec138270 2024-09-13T11:53:12.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.439+0000 7f48eaffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:53:12.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.439+0000 7f48eb7fe640 1 --2- 172.21.15.100:0/967345572 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48ec137e40 0x7f48ec0a4950 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f48ec0a7880 tx=0x7f48d402a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2358cef4a9bbda9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:12.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.439+0000 7f48eaffd640 1 -- 172.21.15.100:0/967345572 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d4037070 con 0x7f48ec137e40 2024-09-13T11:53:12.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.439+0000 7f48eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:12.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.439+0000 7f48eaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:12.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.439+0000 7f48eaffd640 10 monclient: dump: 2024-09-13T11:53:12.452 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:12.452 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:12.452 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.452 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.452 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:12.453 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.453 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:12.453 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:12.453 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:12.453 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.439+0000 7f48eaffd640 10 monclient: _finish_auth 0 2024-09-13T11:53:12.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.439+0000 7f48eaffd640 10 monclient: _check_auth_tickets 2024-09-13T11:53:12.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.439+0000 7f48eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:12.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.439+0000 7f48f291e640 5 monclient: authenticate success, global_id 4382 2024-09-13T11:53:12.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.439+0000 7f48eaffd640 1 -- 172.21.15.100:0/967345572 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48d4032440 con 0x7f48ec137e40 2024-09-13T11:53:12.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.439+0000 7f48f291e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:12.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.439+0000 7f48eaffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:12.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.439+0000 7f48f291e640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:12.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.439+0000 7f48eaffd640 1 -- 172.21.15.100:0/967345572 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d4032810 con 0x7f48ec137e40 2024-09-13T11:53:12.454 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.439+0000 7f48eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:12.454 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.439+0000 7f48f291e640 4 set_mon_vals no callback set 2024-09-13T11:53:12.454 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.439+0000 7f48eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:12.454 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.439+0000 7f48eaffd640 10 monclient: dump: 2024-09-13T11:53:12.454 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:12.454 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:12.454 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.454 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.454 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:12.454 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.454 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:12.454 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:12.454 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:12.454 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.440+0000 7f48f291e640 10 monclient: shutdown 2024-09-13T11:53:12.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.440+0000 7f48f291e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:12.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.440+0000 7f48f291e640 1 -- 172.21.15.100:0/967345572 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48ec137e40 msgr2=0x7f48ec0a4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.440+0000 7f48f291e640 1 --2- 172.21.15.100:0/967345572 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48ec137e40 0x7f48ec0a4950 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f48ec0a7880 tx=0x7f48d402a130 comp rx=0 tx=0).stop 2024-09-13T11:53:12.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.440+0000 7f48f291e640 1 -- 172.21.15.100:0/967345572 shutdown_connections 2024-09-13T11:53:12.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.440+0000 7f48f291e640 1 --2- 172.21.15.100:0/967345572 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48ec137e40 0x7f48ec0a4950 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.440+0000 7f48f291e640 1 --2- 172.21.15.100:0/967345572 >> v2:172.21.15.100:3300/0 conn(0x7f48ec137a60 0x7f48ec0a2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.440+0000 7f48f291e640 1 -- 172.21.15.100:0/967345572 >> 172.21.15.100:0/967345572 conn(0x7f48ec0aa580 msgr2=0x7f48ec0a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:12.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.440+0000 7f48f291e640 1 -- 172.21.15.100:0/967345572 shutdown_connections 2024-09-13T11:53:12.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.440+0000 7f48f291e640 1 -- 172.21.15.100:0/967345572 wait complete. 2024-09-13T11:53:12.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.440+0000 7f48f291e640 10 monclient: build_initial_monmap 2024-09-13T11:53:12.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.440+0000 7f48f291e640 10 monclient: monmap: 2024-09-13T11:53:12.455 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:12.455 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:12.456 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:12.456 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:12.456 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:12.456 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.456 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:12.456 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:12.456 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:12.456 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.441+0000 7f48f291e640 1 Processor -- start 2024-09-13T11:53:12.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.441+0000 7f48f291e640 1 -- start start 2024-09-13T11:53:12.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.441+0000 7f48f291e640 10 monclient: init 2024-09-13T11:53:12.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.441+0000 7f48f291e640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:12.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.441+0000 7f48f291e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:53:12.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.441+0000 7f48f291e640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f48ec137a60 0x7f48ec13ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.441+0000 7f48f291e640 10 monclient(hunting): picked mon.noname-a con 0x7f48ec137a60 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:12.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.441+0000 7f48f291e640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48ec137e40 0x7f48ec1cce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.441+0000 7f48f291e640 10 monclient(hunting): picked mon.noname-c con 0x7f48ec137e40 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:12.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.441+0000 7f48f291e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48ec138270 0x7f48ec13c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.441+0000 7f48f291e640 10 monclient(hunting): picked mon.noname-b con 0x7f48ec138270 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:12.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.441+0000 7f48f291e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.441+0000 7f48f291e640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f48ec06ff00 con 0x7f48ec137a60 2024-09-13T11:53:12.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.441+0000 7f48f291e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.441+0000 7f48f291e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f48ec06fd80 con 0x7f48ec138270 2024-09-13T11:53:12.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.441+0000 7f48f291e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.441+0000 7f48f291e640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f48ec0a55a0 con 0x7f48ec137e40 2024-09-13T11:53:12.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.441+0000 7f48f291e640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:12.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.441+0000 7f48f291e640 10 monclient(hunting): authenticate will time out at 773.035934s 2024-09-13T11:53:12.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48ebfff640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f48ec137a60 0x7f48ec13ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48eb7fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48ec137e40 0x7f48ec1cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48eb7fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48ec137e40 0x7f48ec1cce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:35184/0 (socket says 172.21.15.100:35184) 2024-09-13T11:53:12.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48ebfff640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f48ec137a60 0x7f48ec13ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:36438/0 (socket says 172.21.15.100:36438) 2024-09-13T11:53:12.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48eb7fe640 1 -- 172.21.15.100:0/105109651 learned_addr learned my addr 172.21.15.100:0/105109651 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:12.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48eb7fe640 10 monclient(hunting): get_auth_request con 0x7f48ec137e40 auth_method 0 2024-09-13T11:53:12.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:12.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:12.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:12.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48ebfff640 10 monclient(hunting): get_auth_request con 0x7f48ec137a60 auth_method 0 2024-09-13T11:53:12.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:12.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:12.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:12.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:12.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:12.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:12.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:12.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:12.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:12.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48f0ebb640 1 --2- 172.21.15.100:0/105109651 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48ec138270 0x7f48ec13c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48eb7fe640 10 monclient(hunting): handle_auth_done global_id 4385 payload 824 2024-09-13T11:53:12.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48eb7fe640 1 -- 172.21.15.100:0/105109651 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48ec138270 msgr2=0x7f48ec13c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48eb7fe640 1 --2- 172.21.15.100:0/105109651 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48ec138270 0x7f48ec13c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48eb7fe640 1 -- 172.21.15.100:0/105109651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f48ec137a60 msgr2=0x7f48ec13ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48eb7fe640 1 --2- 172.21.15.100:0/105109651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f48ec137a60 0x7f48ec13ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48eb7fe640 10 monclient: _finish_hunting 0 2024-09-13T11:53:12.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48eb7fe640 1 monclient: found mon.noname-c 2024-09-13T11:53:12.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:12.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48eb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:53:12.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48eb7fe640 1 -- 172.21.15.100:0/105109651 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48ec120c00 con 0x7f48ec137e40 2024-09-13T11:53:12.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48f0ebb640 10 monclient: get_auth_request con 0x7f48ec138270 auth_method 0 2024-09-13T11:53:12.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48f0ebb640 1 --2- 172.21.15.100:0/105109651 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48ec138270 0x7f48ec13c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:53:12.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48eb7fe640 1 --2- 172.21.15.100:0/105109651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48ec137e40 0x7f48ec1cce00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f48d4000c00 tx=0x7f48d40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:12.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48e8ff9640 1 -- 172.21.15.100:0/105109651 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d4041020 con 0x7f48ec137e40 2024-09-13T11:53:12.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:12.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48e8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:12.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48e8ff9640 10 monclient: dump: 2024-09-13T11:53:12.460 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:12.460 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:12.460 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.460 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.461 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:12.461 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.461 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:12.461 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:12.461 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:12.461 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.461 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48e8ff9640 10 monclient: _finish_auth 0 2024-09-13T11:53:12.461 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48e8ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:53:12.461 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:12.461 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48e8ff9640 1 -- 172.21.15.100:0/105109651 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48d4030660 con 0x7f48ec137e40 2024-09-13T11:53:12.461 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48f291e640 5 monclient: authenticate success, global_id 4385 2024-09-13T11:53:12.461 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48f291e640 10 monclient: _renew_subs 2024-09-13T11:53:12.461 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48f291e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:12.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48f291e640 1 -- 172.21.15.100:0/105109651 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48ec120ee0 con 0x7f48ec137e40 2024-09-13T11:53:12.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48e8ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:12.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48e8ff9640 1 -- 172.21.15.100:0/105109651 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d4037040 con 0x7f48ec137e40 2024-09-13T11:53:12.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:12.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48e8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:12.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48e8ff9640 10 monclient: dump: 2024-09-13T11:53:12.462 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:12.462 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:12.462 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.462 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.463 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:12.463 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.463 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:12.463 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:12.463 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:12.463 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.463 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48ea7fc640 4 set_mon_vals no callback set 2024-09-13T11:53:12.463 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48f291e640 10 monclient: _renew_subs 2024-09-13T11:53:12.463 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48f291e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:12.463 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.442+0000 7f48f291e640 1 -- 172.21.15.100:0/105109651 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f48ec0a1140 con 0x7f48ec137e40 2024-09-13T11:53:12.463 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.443+0000 7f48e8ff9640 1 -- 172.21.15.100:0/105109651 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f48d4044020 con 0x7f48ec137e40 2024-09-13T11:53:12.463 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.443+0000 7f48f291e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:12.463 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.443+0000 7f48f291e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:12.463 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.443+0000 7f48f291e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:12.464 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.443+0000 7f48f291e640 1 -- 172.21.15.100:0/105109651 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48ec0a2980 con 0x7f48ec137e40 2024-09-13T11:53:12.464 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.443+0000 7f48e8ff9640 1 --2- 172.21.15.100:0/105109651 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f48cc03dfd0 0x7f48cc040490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.464 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.443+0000 7f48e8ff9640 1 -- 172.21.15.100:0/105109651 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4374+0+0 (secure 0 0 0) 0x7f48d4072d50 con 0x7f48ec137e40 2024-09-13T11:53:12.464 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.444+0000 7f48ebfff640 1 --2- 172.21.15.100:0/105109651 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f48cc03dfd0 0x7f48cc040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.464 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.444+0000 7f48ebfff640 10 monclient: get_auth_request con 0x7f48cc03dfd0 auth_method 0 2024-09-13T11:53:12.464 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.444+0000 7f48ebfff640 1 --2- 172.21.15.100:0/105109651 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f48cc03dfd0 0x7f48cc040490 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f48dc009b40 tx=0x7f48dc009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:12.464 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.448+0000 7f48e8ff9640 1 -- 172.21.15.100:0/105109651 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f48d404b030 con 0x7f48ec137e40 2024-09-13T11:53:12.465 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.448+0000 7f48e8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:12.465 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.448+0000 7f48e8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:12.693 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.690+0000 7f48f291e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:53:12.693 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.690+0000 7f48f291e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:53:12.693 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.690+0000 7f48f291e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:12.693 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.690+0000 7f48f291e640 1 -- 172.21.15.100:0/105109651 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f48ec06a0f0 con 0x7f48ec137e40 2024-09-13T11:53:12.694 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.691+0000 7f48e8ff9640 1 -- 172.21.15.100:0/105109651 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f48d403c380 con 0x7f48ec137e40 2024-09-13T11:53:12.694 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.691+0000 7f48e8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:53:12.694 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.691+0000 7f48e8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:12.694 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":26,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:53:12.696 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.693+0000 7f48f291e640 1 -- 172.21.15.100:0/105109651 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f48cc03dfd0 msgr2=0x7f48cc040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.696 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.693+0000 7f48f291e640 1 --2- 172.21.15.100:0/105109651 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f48cc03dfd0 0x7f48cc040490 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f48dc009b40 tx=0x7f48dc009190 comp rx=0 tx=0).stop 2024-09-13T11:53:12.696 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.693+0000 7f48f291e640 10 monclient: shutdown 2024-09-13T11:53:12.696 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.693+0000 7f48f291e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:12.696 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.693+0000 7f48f291e640 1 -- 172.21.15.100:0/105109651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48ec137e40 msgr2=0x7f48ec1cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.696 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.693+0000 7f48f291e640 1 --2- 172.21.15.100:0/105109651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48ec137e40 0x7f48ec1cce00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f48d4000c00 tx=0x7f48d40046c0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.696 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.693+0000 7f48f291e640 1 -- 172.21.15.100:0/105109651 shutdown_connections 2024-09-13T11:53:12.696 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.693+0000 7f48f291e640 1 --2- 172.21.15.100:0/105109651 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f48cc03dfd0 0x7f48cc040490 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.696 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.693+0000 7f48f291e640 1 --2- 172.21.15.100:0/105109651 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48ec138270 0x7f48ec13c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.696 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.693+0000 7f48f291e640 1 --2- 172.21.15.100:0/105109651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f48ec137e40 0x7f48ec1cce00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.696 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.693+0000 7f48f291e640 1 --2- 172.21.15.100:0/105109651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f48ec137a60 0x7f48ec13ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.697 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.693+0000 7f48f291e640 1 -- 172.21.15.100:0/105109651 >> 172.21.15.100:0/105109651 conn(0x7f48ec0aa580 msgr2=0x7f48ec0a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:12.697 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.693+0000 7f48f291e640 1 -- 172.21.15.100:0/105109651 shutdown_connections 2024-09-13T11:53:12.697 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.693+0000 7f48f291e640 1 -- 172.21.15.100:0/105109651 wait complete. 2024-09-13T11:53:12.707 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-13T11:53:12.707 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-13T11:53:12.707 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-13T11:53:12.707 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-13T11:53:12.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.801+0000 7f1989144640 10 monclient: get_monmap_and_config 2024-09-13T11:53:12.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.801+0000 7f1989144640 10 monclient: build_initial_monmap 2024-09-13T11:53:12.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.801+0000 7f1989144640 10 monclient: monmap: 2024-09-13T11:53:12.804 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:12.804 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:12.804 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:12.803733+0000 2024-09-13T11:53:12.805 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:12.803733+0000 2024-09-13T11:53:12.805 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:12.805 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.805 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:12.805 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:12.805 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:12.805 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:12.805 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:12.805 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.802+0000 7f1989144640 1 Processor -- start 2024-09-13T11:53:12.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.802+0000 7f1989144640 1 -- start start 2024-09-13T11:53:12.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.802+0000 7f1989144640 10 monclient: init 2024-09-13T11:53:12.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.802+0000 7f1989144640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:12.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1989144640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-13T11:53:12.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1989144640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1984137870 0x7f198409ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1989144640 10 monclient(hunting): picked mon.noname-b con 0x7f1984137870 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:12.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1989144640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1984137c50 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:12.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1989144640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1984138050 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:12.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1989144640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19840a44e0 con 0x7f1984137c50 2024-09-13T11:53:12.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1989144640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19840a3460 con 0x7f1984138050 2024-09-13T11:53:12.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1989144640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1989144640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f198409c6c0 con 0x7f1984137870 2024-09-13T11:53:12.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1989144640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:12.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1989144640 10 monclient(hunting): authenticate will time out at 773.397082s 2024-09-13T11:53:12.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1982d9d640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1984137870 0x7f198409ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f198259c640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f1984137c50 0x7f19840a2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:34160/0 (socket says 172.21.15.100:34160) 2024-09-13T11:53:12.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f198259c640 1 -- 172.21.15.100:0/1053317109 learned_addr learned my addr 172.21.15.100:0/1053317109 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:53:12.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1982d9d640 10 monclient(hunting): get_auth_request con 0x7f1984137870 auth_method 0 2024-09-13T11:53:12.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1982d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:12.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1982d9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:12.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1982d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:12.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1982d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:12.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1982d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:12.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1982d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:12.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1982d9d640 10 monclient(hunting): handle_auth_done global_id 4388 payload 274 2024-09-13T11:53:12.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1982d9d640 1 -- 172.21.15.100:0/1053317109 >> v1:172.21.15.122:6789/0 conn(0x7f1984138050 legacy=0x7f19840a2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:12.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1981d9b640 1 -- 172.21.15.100:0/1053317109 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2315666851 0 0) 0x7f19840a44e0 con 0x7f1984137c50 2024-09-13T11:53:12.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1982d9d640 1 -- 172.21.15.100:0/1053317109 >> v1:172.21.15.100:6789/0 conn(0x7f1984137c50 legacy=0x7f19840a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1982d9d640 10 monclient: _finish_hunting 0 2024-09-13T11:53:12.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1982d9d640 1 monclient: found mon.noname-b 2024-09-13T11:53:12.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1982d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:12.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1982d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:53:12.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1982d9d640 1 -- 172.21.15.100:0/1053317109 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1984076cc0 con 0x7f1984137870 2024-09-13T11:53:12.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.803+0000 7f1981d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:53:12.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1982d9d640 1 --2- 172.21.15.100:0/1053317109 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1984137870 0x7f198409ca50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1974009dc0 tx=0x7f197402a510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5fe31b3aeec41e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:12.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1981d9b640 1 -- 172.21.15.100:0/1053317109 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1974007740 con 0x7f1984137870 2024-09-13T11:53:12.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1981d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:12.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1981d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:12.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1981d9b640 10 monclient: dump: 2024-09-13T11:53:12.817 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:12.817 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:12.817 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.817 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.817 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:12.817 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.817 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:12.817 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:12.818 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:12.818 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1981d9b640 10 monclient: _finish_auth 0 2024-09-13T11:53:12.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1981d9b640 10 monclient: _check_auth_tickets 2024-09-13T11:53:12.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1981d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:12.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1989144640 5 monclient: authenticate success, global_id 4388 2024-09-13T11:53:12.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1981d9b640 1 -- 172.21.15.100:0/1053317109 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19740078e0 con 0x7f1984137870 2024-09-13T11:53:12.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1989144640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:12.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1981d9b640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:12.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1989144640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:12.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1989144640 4 set_mon_vals no callback set 2024-09-13T11:53:12.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1981d9b640 1 -- 172.21.15.100:0/1053317109 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1974007cb0 con 0x7f1984137870 2024-09-13T11:53:12.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1981d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:12.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1981d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:12.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1981d9b640 10 monclient: dump: 2024-09-13T11:53:12.819 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:12.819 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:12.819 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.819 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.819 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:12.819 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.819 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:12.819 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:12.819 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:12.819 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1989144640 10 monclient: shutdown 2024-09-13T11:53:12.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1989144640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:12.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1989144640 1 -- 172.21.15.100:0/1053317109 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1984137870 msgr2=0x7f198409ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1989144640 1 --2- 172.21.15.100:0/1053317109 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1984137870 0x7f198409ca50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1974009dc0 tx=0x7f197402a510 comp rx=0 tx=0).stop 2024-09-13T11:53:12.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1989144640 1 -- 172.21.15.100:0/1053317109 shutdown_connections 2024-09-13T11:53:12.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1989144640 1 --2- 172.21.15.100:0/1053317109 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1984137870 0x7f198409ca50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1989144640 1 -- 172.21.15.100:0/1053317109 >> 172.21.15.100:0/1053317109 conn(0x7f19840a7260 msgr2=0x7f19840a3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:12.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1989144640 1 -- 172.21.15.100:0/1053317109 shutdown_connections 2024-09-13T11:53:12.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.804+0000 7f1989144640 1 -- 172.21.15.100:0/1053317109 wait complete. 2024-09-13T11:53:12.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.805+0000 7f1989144640 10 monclient: build_initial_monmap 2024-09-13T11:53:12.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.805+0000 7f1989144640 10 monclient: monmap: 2024-09-13T11:53:12.820 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:12.820 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:12.820 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:12.820 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:12.821 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:12.821 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.821 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:12.821 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:12.821 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:12.821 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1989144640 1 Processor -- start 2024-09-13T11:53:12.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1989144640 1 -- start start 2024-09-13T11:53:12.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1989144640 10 monclient: init 2024-09-13T11:53:12.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1989144640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:12.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1989144640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:53:12.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1989144640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1984137870 0x7f198411d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1989144640 10 monclient(hunting): picked mon.noname-c con 0x7f1984137870 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:12.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1989144640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1984137c50 0x7f198411d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1989144640 10 monclient(hunting): picked mon.noname-a con 0x7f1984137c50 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:12.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1989144640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1984138050 0x7f198411dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1989144640 10 monclient(hunting): picked mon.noname-b con 0x7f1984138050 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:12.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1989144640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1989144640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f1984074690 con 0x7f1984137c50 2024-09-13T11:53:12.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1989144640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1989144640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1984074510 con 0x7f1984138050 2024-09-13T11:53:12.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1989144640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1989144640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f1984140690 con 0x7f1984137870 2024-09-13T11:53:12.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1989144640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:12.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1989144640 10 monclient(hunting): authenticate will time out at 773.400514s 2024-09-13T11:53:12.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1982d9d640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1984137870 0x7f198411d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f198259c640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1984137c50 0x7f198411d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1982d9d640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1984137870 0x7f198411d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:35206/0 (socket says 172.21.15.100:35206) 2024-09-13T11:53:12.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1982d9d640 1 -- 172.21.15.100:0/3613358161 learned_addr learned my addr 172.21.15.100:0/3613358161 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:12.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1982d9d640 10 monclient(hunting): get_auth_request con 0x7f1984137870 auth_method 0 2024-09-13T11:53:12.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1982d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:12.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1982d9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:12.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1982d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:12.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f198259c640 10 monclient(hunting): get_auth_request con 0x7f1984137c50 auth_method 0 2024-09-13T11:53:12.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f198259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:12.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f198259c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:12.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f198259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:12.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1982d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:12.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1982d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:12.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1982d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:12.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f198259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:12.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f198259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:12.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f198259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:12.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f198359e640 1 --2- 172.21.15.100:0/3613358161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1984138050 0x7f198411dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f198359e640 10 monclient(hunting): get_auth_request con 0x7f1984138050 auth_method 0 2024-09-13T11:53:12.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f198359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:12.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f198359e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:12.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f198359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:12.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1982d9d640 10 monclient(hunting): handle_auth_done global_id 4391 payload 824 2024-09-13T11:53:12.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1982d9d640 1 -- 172.21.15.100:0/3613358161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1984138050 msgr2=0x7f198411dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1982d9d640 1 --2- 172.21.15.100:0/3613358161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1984138050 0x7f198411dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1982d9d640 1 -- 172.21.15.100:0/3613358161 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1984137c50 msgr2=0x7f198411d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1982d9d640 1 --2- 172.21.15.100:0/3613358161 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1984137c50 0x7f198411d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1982d9d640 10 monclient: _finish_hunting 0 2024-09-13T11:53:12.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1982d9d640 1 monclient: found mon.noname-c 2024-09-13T11:53:12.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1982d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:12.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1982d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:53:12.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f1982d9d640 1 -- 172.21.15.100:0/3613358161 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f198411f8d0 con 0x7f1984137870 2024-09-13T11:53:12.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.806+0000 7f198259c640 1 --2- 172.21.15.100:0/3613358161 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1984137c50 0x7f198411d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:12.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.807+0000 7f1982d9d640 1 --2- 172.21.15.100:0/3613358161 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1984137870 0x7f198411d280 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1974000f80 tx=0x7f1974004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:12.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.807+0000 7f196b7fe640 1 -- 172.21.15.100:0/3613358161 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1974040070 con 0x7f1984137870 2024-09-13T11:53:12.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.807+0000 7f196b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:12.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.807+0000 7f196b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:12.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.807+0000 7f196b7fe640 10 monclient: dump: 2024-09-13T11:53:12.826 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:12.826 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:12.826 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.826 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.826 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:12.826 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.826 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:12.826 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:12.826 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:12.827 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.807+0000 7f196b7fe640 10 monclient: _finish_auth 0 2024-09-13T11:53:12.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.807+0000 7f196b7fe640 10 monclient: _check_auth_tickets 2024-09-13T11:53:12.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.807+0000 7f196b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:12.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.807+0000 7f196b7fe640 1 -- 172.21.15.100:0/3613358161 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1974031d00 con 0x7f1984137870 2024-09-13T11:53:12.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.807+0000 7f1989144640 5 monclient: authenticate success, global_id 4391 2024-09-13T11:53:12.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.807+0000 7f1989144640 10 monclient: _renew_subs 2024-09-13T11:53:12.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.807+0000 7f1989144640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:12.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.807+0000 7f1989144640 1 -- 172.21.15.100:0/3613358161 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f198411fbb0 con 0x7f1984137870 2024-09-13T11:53:12.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.807+0000 7f196b7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:12.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.807+0000 7f196b7fe640 1 -- 172.21.15.100:0/3613358161 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1974045680 con 0x7f1984137870 2024-09-13T11:53:12.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.807+0000 7f196b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:12.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.807+0000 7f196b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:12.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.807+0000 7f196b7fe640 10 monclient: dump: 2024-09-13T11:53:12.828 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:12.828 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:12.828 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.828 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.828 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:12.828 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.828 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:12.828 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:12.828 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:12.828 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.807+0000 7f198159a640 4 set_mon_vals no callback set 2024-09-13T11:53:12.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.807+0000 7f1989144640 10 monclient: _renew_subs 2024-09-13T11:53:12.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.807+0000 7f1989144640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:12.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.807+0000 7f1989144640 1 -- 172.21.15.100:0/3613358161 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f198413f4b0 con 0x7f1984137870 2024-09-13T11:53:12.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.808+0000 7f1989144640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:12.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.808+0000 7f1989144640 1 --2- 172.21.15.100:0/3613358161 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19841cce70 0x7f198402d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.808+0000 7f1989144640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.808+0000 7f1989144640 1 -- 172.21.15.100:0/3613358161 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f198411e3b0 con 0x7f19841cce70 2024-09-13T11:53:12.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.808+0000 7f198259c640 1 --2- 172.21.15.100:0/3613358161 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19841cce70 0x7f198402d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.808+0000 7f198259c640 10 monclient: get_auth_request con 0x7f19841cce70 auth_method 0 2024-09-13T11:53:12.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.808+0000 7f198259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:12.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.808+0000 7f198259c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:12.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.808+0000 7f198259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:12.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.808+0000 7f196b7fe640 1 -- 172.21.15.100:0/3613358161 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f1974039040 con 0x7f1984137870 2024-09-13T11:53:12.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.808+0000 7f198259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:12.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.808+0000 7f198259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:12.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.808+0000 7f198259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:12.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.808+0000 7f198259c640 10 monclient(hunting): handle_auth_done global_id 4440 payload 824 2024-09-13T11:53:12.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.808+0000 7f198259c640 1 -- 172.21.15.100:0/3613358161 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f19840a65c0 con 0x7f19841cce70 2024-09-13T11:53:12.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.808+0000 7f196b7fe640 1 --2- 172.21.15.100:0/3613358161 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f197003dfa0 0x7f1970040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.808+0000 7f196b7fe640 1 -- 172.21.15.100:0/3613358161 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4374+0+0 (secure 0 0 0) 0x7f1974074180 con 0x7f1984137870 2024-09-13T11:53:12.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.808+0000 7f198259c640 1 --2- 172.21.15.100:0/3613358161 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19841cce70 0x7f198402d1f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1978000f60 tx=0x7f1978001d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:12.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.808+0000 7f196b7fe640 1 -- 172.21.15.100:0/3613358161 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197800f9d0 con 0x7f19841cce70 2024-09-13T11:53:12.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.808+0000 7f196b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:12.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.808+0000 7f196b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:12.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.808+0000 7f198359e640 1 --2- 172.21.15.100:0/3613358161 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f197003dfa0 0x7f1970040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.808+0000 7f196b7fe640 10 monclient: dump: 2024-09-13T11:53:12.832 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:12.832 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:12.832 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.832 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.832 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:12.832 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.832 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:12.833 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:12.833 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:12.833 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.808+0000 7f198359e640 10 monclient: get_auth_request con 0x7f197003dfa0 auth_method 0 2024-09-13T11:53:12.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.809+0000 7f196b7fe640 1 -- 172.21.15.100:0/3613358161 <== mon.0 v2:172.21.15.100:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1978018070 con 0x7f19841cce70 2024-09-13T11:53:12.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.809+0000 7f196b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:12.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.809+0000 7f196b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:12.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.809+0000 7f196b7fe640 1 -- 172.21.15.100:0/3613358161 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19841cce70 msgr2=0x7f198402d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.809+0000 7f196b7fe640 1 --2- 172.21.15.100:0/3613358161 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19841cce70 0x7f198402d1f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1978000f60 tx=0x7f1978001d60 comp rx=0 tx=0).stop 2024-09-13T11:53:12.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.809+0000 7f196b7fe640 1 -- 172.21.15.100:0/3613358161 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19841cce70 msgr2=0x7f198402d1f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:53:12.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.809+0000 7f196b7fe640 1 --2- 172.21.15.100:0/3613358161 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19841cce70 0x7f198402d1f0 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1978000f60 tx=0x7f1978001d60 comp rx=0 tx=0).stop 2024-09-13T11:53:12.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.809+0000 7f198359e640 1 --2- 172.21.15.100:0/3613358161 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f197003dfa0 0x7f1970040460 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f198409d570 tx=0x7f197c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:12.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f1989144640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:53:12.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f1989144640 1 --2- 172.21.15.100:0/3613358161 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19841cd250 0x7f198402d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f1989144640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f1989144640 1 -- 172.21.15.100:0/3613358161 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f19840a06a0 con 0x7f19841cd250 2024-09-13T11:53:12.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f198259c640 1 --2- 172.21.15.100:0/3613358161 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19841cd250 0x7f198402d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f198259c640 10 monclient: get_auth_request con 0x7f19841cd250 auth_method 0 2024-09-13T11:53:12.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f198259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:12.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f198259c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:12.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f198259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:12.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f198259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:12.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f198259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:12.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f198259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:12.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f198259c640 10 monclient(hunting): handle_auth_done global_id 4443 payload 824 2024-09-13T11:53:12.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f198259c640 1 -- 172.21.15.100:0/3613358161 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1984074f70 con 0x7f19841cd250 2024-09-13T11:53:12.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f198259c640 1 --2- 172.21.15.100:0/3613358161 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19841cd250 0x7f198402d730 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f198413f210 tx=0x7f1978001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:12.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f196b7fe640 1 -- 172.21.15.100:0/3613358161 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197801ee60 con 0x7f19841cd250 2024-09-13T11:53:12.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f196b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:12.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f196b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:12.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f196b7fe640 10 monclient: dump: 2024-09-13T11:53:12.842 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:12.842 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:12.842 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.842 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.842 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:12.843 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.843 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:12.843 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:12.843 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:12.843 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f196b7fe640 1 -- 172.21.15.100:0/3613358161 <== mon.0 v2:172.21.15.100:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4721 (secure 0 0 0) 0x7f1978013600 con 0x7f19841cd250 2024-09-13T11:53:12.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f196b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:53:12.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f196b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:12.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f196b7fe640 1 -- 172.21.15.100:0/3613358161 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19841cd250 msgr2=0x7f198402d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f196b7fe640 1 --2- 172.21.15.100:0/3613358161 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19841cd250 0x7f198402d730 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f198413f210 tx=0x7f1978001080 comp rx=0 tx=0).stop 2024-09-13T11:53:12.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f196b7fe640 1 -- 172.21.15.100:0/3613358161 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19841cd250 msgr2=0x7f198402d730 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:53:12.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.815+0000 7f196b7fe640 1 --2- 172.21.15.100:0/3613358161 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19841cd250 0x7f198402d730 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f198413f210 tx=0x7f1978001080 comp rx=0 tx=0).stop 2024-09-13T11:53:12.845 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:53:12.845 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:53:12.845 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:53:12.845 INFO:teuthology.orchestra.run.smithi100.stdout: "state": "leader", 2024-09-13T11:53:12.846 INFO:teuthology.orchestra.run.smithi100.stdout: "election_epoch": 8, 2024-09-13T11:53:12.846 INFO:teuthology.orchestra.run.smithi100.stdout: "uptime": 31886, 2024-09-13T11:53:12.846 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum": [ 2024-09-13T11:53:12.846 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-13T11:53:12.846 INFO:teuthology.orchestra.run.smithi100.stdout: 1, 2024-09-13T11:53:12.846 INFO:teuthology.orchestra.run.smithi100.stdout: 2 2024-09-13T11:53:12.847 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:12.847 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_age": 26, 2024-09-13T11:53:12.847 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:53:12.847 INFO:teuthology.orchestra.run.smithi100.stdout: "required_con": "2450521708007866372", 2024-09-13T11:53:12.847 INFO:teuthology.orchestra.run.smithi100.stdout: "required_mon": [ 2024-09-13T11:53:12.847 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:53:12.848 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:53:12.848 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:53:12.848 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:53:12.848 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:53:12.848 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:53:12.848 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:53:12.849 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:53:12.849 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:53:12.849 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:53:12.849 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:53:12.849 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:12.849 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:53:12.850 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_mon": [ 2024-09-13T11:53:12.850 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:53:12.850 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:53:12.850 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:53:12.850 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:53:12.850 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:53:12.851 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:53:12.851 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:53:12.851 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:53:12.851 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:53:12.851 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:53:12.851 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:53:12.852 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:12.852 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:12.852 INFO:teuthology.orchestra.run.smithi100.stdout: "outside_quorum": [], 2024-09-13T11:53:12.852 INFO:teuthology.orchestra.run.smithi100.stdout: "extra_probe_peers": [], 2024-09-13T11:53:12.852 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_provider": [], 2024-09-13T11:53:12.852 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": { 2024-09-13T11:53:12.853 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-09-13T11:53:12.853 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "0b1f4b4d-fad4-4dd3-86ac-a286d80846c0", 2024-09-13T11:53:12.853 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:53:12.853 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:53:12.853 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release": 19, 2024-09-13T11:53:12.853 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release_name": "squid", 2024-09-13T11:53:12.854 INFO:teuthology.orchestra.run.smithi100.stdout: "election_strategy": 1, 2024-09-13T11:53:12.854 INFO:teuthology.orchestra.run.smithi100.stdout: "disallowed_leaders: ": "", 2024-09-13T11:53:12.854 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false, 2024-09-13T11:53:12.854 INFO:teuthology.orchestra.run.smithi100.stdout: "tiebreaker_mon": "", 2024-09-13T11:53:12.854 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_ranks: ": "", 2024-09-13T11:53:12.854 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:53:12.854 INFO:teuthology.orchestra.run.smithi100.stdout: "persistent": [ 2024-09-13T11:53:12.854 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:53:12.854 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:53:12.854 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:53:12.854 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:53:12.854 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:53:12.854 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:53:12.855 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:53:12.855 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:53:12.855 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:53:12.855 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:53:12.855 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:53:12.855 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:12.855 INFO:teuthology.orchestra.run.smithi100.stdout: "optional": [] 2024-09-13T11:53:12.855 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:12.855 INFO:teuthology.orchestra.run.smithi100.stdout: "mons": [ 2024-09-13T11:53:12.855 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:12.855 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:53:12.855 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:53:12.855 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:53:12.855 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:53:12.856 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:12.856 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:53:12.856 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3300", 2024-09-13T11:53:12.856 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:12.856 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:12.856 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:12.856 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:53:12.856 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789", 2024-09-13T11:53:12.856 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:12.856 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:12.856 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:12.856 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:12.856 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789/0", 2024-09-13T11:53:12.856 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-13T11:53:12.857 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:53:12.857 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:53:12.857 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:53:12.857 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:12.857 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:12.857 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:53:12.857 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:53:12.857 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:53:12.857 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:53:12.857 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:12.857 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:53:12.857 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:3300", 2024-09-13T11:53:12.857 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:12.857 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:12.858 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:12.858 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:53:12.858 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789", 2024-09-13T11:53:12.858 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:12.858 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:12.858 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:12.858 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:12.858 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789/0", 2024-09-13T11:53:12.858 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.122:6789/0", 2024-09-13T11:53:12.858 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:53:12.858 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:53:12.858 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:53:12.858 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:12.858 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:12.859 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:53:12.859 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:53:12.859 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:53:12.859 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:53:12.859 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:12.859 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:53:12.859 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3301", 2024-09-13T11:53:12.859 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:12.859 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:12.859 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:12.859 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:53:12.859 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790", 2024-09-13T11:53:12.859 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:12.859 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:12.860 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:12.860 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:12.860 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790/0", 2024-09-13T11:53:12.860 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-13T11:53:12.860 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:53:12.860 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:53:12.860 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:53:12.860 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:12.860 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:12.860 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:12.860 INFO:teuthology.orchestra.run.smithi100.stdout: "feature_map": { 2024-09-13T11:53:12.860 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": [ 2024-09-13T11:53:12.860 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:12.860 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:53:12.861 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:53:12.861 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:53:12.861 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:12.861 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:12.861 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": [ 2024-09-13T11:53:12.861 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:12.861 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:53:12.861 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:53:12.861 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:53:12.861 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:12.861 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:12.861 INFO:teuthology.orchestra.run.smithi100.stdout: "client": [ 2024-09-13T11:53:12.861 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:12.861 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:53:12.861 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:53:12.862 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 2 2024-09-13T11:53:12.862 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:12.862 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:12.862 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:12.862 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false 2024-09-13T11:53:12.862 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:53:12.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.816+0000 7f1989144640 1 -- 172.21.15.100:0/3613358161 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f197003dfa0 msgr2=0x7f1970040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.816+0000 7f1989144640 1 --2- 172.21.15.100:0/3613358161 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f197003dfa0 0x7f1970040460 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f198409d570 tx=0x7f197c009040 comp rx=0 tx=0).stop 2024-09-13T11:53:12.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.816+0000 7f1989144640 10 monclient: shutdown 2024-09-13T11:53:12.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.816+0000 7f1989144640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:12.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.816+0000 7f1989144640 1 -- 172.21.15.100:0/3613358161 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1984137870 msgr2=0x7f198411d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.816+0000 7f1989144640 1 --2- 172.21.15.100:0/3613358161 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1984137870 0x7f198411d280 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1974000f80 tx=0x7f1974004a40 comp rx=0 tx=0).stop 2024-09-13T11:53:12.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.816+0000 7f198359e640 1 -- 172.21.15.100:0/3613358161 reap_dead start 2024-09-13T11:53:12.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.816+0000 7f1989144640 1 -- 172.21.15.100:0/3613358161 shutdown_connections 2024-09-13T11:53:12.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.816+0000 7f1989144640 1 --2- 172.21.15.100:0/3613358161 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1984137870 0x7f198411d280 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.816+0000 7f1989144640 1 -- 172.21.15.100:0/3613358161 >> 172.21.15.100:0/3613358161 conn(0x7f19840a7260 msgr2=0x7f198409ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:12.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.816+0000 7f1989144640 1 -- 172.21.15.100:0/3613358161 shutdown_connections 2024-09-13T11:53:12.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.816+0000 7f1989144640 1 -- 172.21.15.100:0/3613358161 wait complete. 2024-09-13T11:53:12.865 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T11:53:12.865 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-13T11:53:12.865 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T11:53:12.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.961+0000 7fdc227c3640 10 monclient: get_monmap_and_config 2024-09-13T11:53:12.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.961+0000 7fdc227c3640 10 monclient: build_initial_monmap 2024-09-13T11:53:12.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.961+0000 7fdc227c3640 10 monclient: monmap: 2024-09-13T11:53:12.964 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:12.964 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:12.964 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:12.962900+0000 2024-09-13T11:53:12.964 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:12.962900+0000 2024-09-13T11:53:12.964 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:12.964 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.964 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:12.964 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:12.964 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:12.964 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:12.965 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:12.965 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.975 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.961+0000 7fdc227c3640 1 Processor -- start 2024-09-13T11:53:12.975 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.961+0000 7fdc227c3640 1 -- start start 2024-09-13T11:53:12.975 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.961+0000 7fdc227c3640 10 monclient: init 2024-09-13T11:53:12.975 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc227c3640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:12.975 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc227c3640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-13T11:53:12.975 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc227c3640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c0a15a0 0x7fdc1c0a1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.975 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc227c3640 10 monclient(hunting): picked mon.noname-b con 0x7fdc1c0a15a0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:12.975 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc227c3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdc1c09f6d0 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:12.975 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc227c3640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fdc1c0a1ec0 0x7fdc1c134ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc227c3640 10 monclient(hunting): picked mon.noname-a con 0x7fdc1c0a1ec0 addr v2:172.21.15.100:3300/0 2024-09-13T11:53:12.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc227c3640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc1c09ecf0 con 0x7fdc1c09f6d0 2024-09-13T11:53:12.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc227c3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc227c3640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7fdc1c09d3b0 con 0x7fdc1c0a1ec0 2024-09-13T11:53:12.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc227c3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc227c3640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fdc1c09ea10 con 0x7fdc1c0a15a0 2024-09-13T11:53:12.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc227c3640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:12.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc227c3640 10 monclient(hunting): authenticate will time out at 773.556388s 2024-09-13T11:53:12.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc1bfff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c0a15a0 0x7fdc1c0a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc20d60640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fdc1c0a1ec0 0x7fdc1c134ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc1b7fe640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fdc1c09f6d0 0x7fdc1c1343d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:34168/0 (socket says 172.21.15.100:34168) 2024-09-13T11:53:12.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc1bfff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c0a15a0 0x7fdc1c0a1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:35218/0 (socket says 172.21.15.100:35218) 2024-09-13T11:53:12.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc20d60640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fdc1c0a1ec0 0x7fdc1c134ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:36484/0 (socket says 172.21.15.100:36484) 2024-09-13T11:53:12.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc1b7fe640 1 -- 172.21.15.100:0/3644023845 learned_addr learned my addr 172.21.15.100:0/3644023845 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:53:12.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc1bfff640 10 monclient(hunting): get_auth_request con 0x7fdc1c0a15a0 auth_method 0 2024-09-13T11:53:12.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:12.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:12.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:12.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc20d60640 10 monclient(hunting): get_auth_request con 0x7fdc1c0a1ec0 auth_method 0 2024-09-13T11:53:12.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc20d60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:12.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc20d60640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:12.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc20d60640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:12.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc20d60640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:12.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc20d60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:12.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc20d60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:12.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:12.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:12.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:12.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc20d60640 10 monclient(hunting): handle_auth_done global_id 4446 payload 274 2024-09-13T11:53:12.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc20d60640 1 -- 172.21.15.100:0/3644023845 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c0a15a0 msgr2=0x7fdc1c0a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:12.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc20d60640 1 --2- 172.21.15.100:0/3644023845 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c0a15a0 0x7fdc1c0a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc20d60640 1 -- 172.21.15.100:0/3644023845 >> v1:172.21.15.100:6789/0 conn(0x7fdc1c09f6d0 legacy=0x7fdc1c1343d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc20d60640 10 monclient: _finish_hunting 0 2024-09-13T11:53:12.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc20d60640 1 monclient: found mon.noname-a 2024-09-13T11:53:12.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc20d60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:12.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc20d60640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:12.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc20d60640 1 -- 172.21.15.100:0/3644023845 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc1c0a2350 con 0x7fdc1c0a1ec0 2024-09-13T11:53:12.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc1affd640 1 -- 172.21.15.100:0/3644023845 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 980435977 0 0) 0x7fdc1c09ecf0 con 0x7fdc1c09f6d0 2024-09-13T11:53:12.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc1affd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:53:12.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.962+0000 7fdc1bfff640 1 --2- 172.21.15.100:0/3644023845 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c0a15a0 0x7fdc1c0a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:12.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.963+0000 7fdc20d60640 1 --2- 172.21.15.100:0/3644023845 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc1c0a1ec0 0x7fdc1c134ae0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdc1400f370 tx=0x7fdc1402e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88fe8ea35c953dd4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:12.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.963+0000 7fdc1affd640 1 -- 172.21.15.100:0/3644023845 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1400d4e0 con 0x7fdc1c0a1ec0 2024-09-13T11:53:12.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.963+0000 7fdc1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:12.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.963+0000 7fdc1affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:12.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.963+0000 7fdc1affd640 10 monclient: dump: 2024-09-13T11:53:12.979 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:12.979 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:12.979 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.980 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.980 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:12.980 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.980 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:12.980 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:12.980 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:12.980 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.980 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.963+0000 7fdc1affd640 10 monclient: _finish_auth 0 2024-09-13T11:53:12.980 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.963+0000 7fdc1affd640 10 monclient: _check_auth_tickets 2024-09-13T11:53:12.980 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.963+0000 7fdc1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:12.980 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.963+0000 7fdc1affd640 1 -- 172.21.15.100:0/3644023845 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc14039070 con 0x7fdc1c0a1ec0 2024-09-13T11:53:12.980 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.963+0000 7fdc227c3640 5 monclient: authenticate success, global_id 4446 2024-09-13T11:53:12.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.963+0000 7fdc227c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:12.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.963+0000 7fdc1affd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:12.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.963+0000 7fdc1affd640 1 -- 172.21.15.100:0/3644023845 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1400b960 con 0x7fdc1c0a1ec0 2024-09-13T11:53:12.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.963+0000 7fdc1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:12.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.963+0000 7fdc1affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:12.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.963+0000 7fdc1affd640 10 monclient: dump: 2024-09-13T11:53:12.981 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:12.981 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:12.981 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.982 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.982 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:12.982 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.982 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:12.982 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:12.982 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:12.982 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.963+0000 7fdc227c3640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:12.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.963+0000 7fdc227c3640 4 set_mon_vals no callback set 2024-09-13T11:53:12.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.964+0000 7fdc227c3640 10 monclient: shutdown 2024-09-13T11:53:12.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.964+0000 7fdc227c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:12.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.964+0000 7fdc227c3640 1 -- 172.21.15.100:0/3644023845 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc1c0a1ec0 msgr2=0x7fdc1c134ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.964+0000 7fdc227c3640 1 --2- 172.21.15.100:0/3644023845 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc1c0a1ec0 0x7fdc1c134ae0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdc1400f370 tx=0x7fdc1402e960 comp rx=0 tx=0).stop 2024-09-13T11:53:12.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.964+0000 7fdc227c3640 1 -- 172.21.15.100:0/3644023845 shutdown_connections 2024-09-13T11:53:12.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.964+0000 7fdc227c3640 1 --2- 172.21.15.100:0/3644023845 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc1c0a1ec0 0x7fdc1c134ae0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.964+0000 7fdc227c3640 1 --2- 172.21.15.100:0/3644023845 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c0a15a0 0x7fdc1c0a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.964+0000 7fdc227c3640 1 -- 172.21.15.100:0/3644023845 >> 172.21.15.100:0/3644023845 conn(0x7fdc1c0a6470 msgr2=0x7fdc1c0a2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:12.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.964+0000 7fdc227c3640 1 -- 172.21.15.100:0/3644023845 shutdown_connections 2024-09-13T11:53:12.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.964+0000 7fdc227c3640 1 -- 172.21.15.100:0/3644023845 wait complete. 2024-09-13T11:53:12.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.964+0000 7fdc227c3640 10 monclient: build_initial_monmap 2024-09-13T11:53:12.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.964+0000 7fdc227c3640 10 monclient: monmap: 2024-09-13T11:53:12.984 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:12.984 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:12.984 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:12.985 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:12.985 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:12.985 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.985 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:12.985 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:12.985 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:12.985 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc227c3640 1 Processor -- start 2024-09-13T11:53:12.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc227c3640 1 -- start start 2024-09-13T11:53:12.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc227c3640 10 monclient: init 2024-09-13T11:53:12.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc227c3640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:12.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc227c3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:53:12.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc227c3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc1c09f6d0 0x7fdc1c121de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc227c3640 10 monclient(hunting): picked mon.noname-b con 0x7fdc1c09f6d0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:12.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc227c3640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c0a15a0 0x7fdc1c11f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc227c3640 10 monclient(hunting): picked mon.noname-c con 0x7fdc1c0a15a0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:12.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc227c3640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc1c0a1ec0 0x7fdc1c11fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc227c3640 10 monclient(hunting): picked mon.noname-a con 0x7fdc1c0a1ec0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:12.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc227c3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc227c3640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c1351a0 con 0x7fdc1c0a1ec0 2024-09-13T11:53:12.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc227c3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc227c3640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c135020 con 0x7fdc1c09f6d0 2024-09-13T11:53:12.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc227c3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc227c3640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fdc1c09cfa0 con 0x7fdc1c0a15a0 2024-09-13T11:53:12.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc227c3640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:12.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc227c3640 10 monclient(hunting): authenticate will time out at 773.559851s 2024-09-13T11:53:12.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc1b7fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c0a15a0 0x7fdc1c11f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc20d60640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc1c0a1ec0 0x7fdc1c11fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc20d60640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc1c0a1ec0 0x7fdc1c11fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:36490/0 (socket says 172.21.15.100:36490) 2024-09-13T11:53:12.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc1b7fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c0a15a0 0x7fdc1c11f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:35230/0 (socket says 172.21.15.100:35230) 2024-09-13T11:53:12.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc20d60640 1 -- 172.21.15.100:0/3002018217 learned_addr learned my addr 172.21.15.100:0/3002018217 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:12.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc20d60640 10 monclient(hunting): get_auth_request con 0x7fdc1c0a1ec0 auth_method 0 2024-09-13T11:53:12.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc20d60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:12.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc20d60640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:12.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc20d60640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:12.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc1b7fe640 10 monclient(hunting): get_auth_request con 0x7fdc1c0a15a0 auth_method 0 2024-09-13T11:53:12.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:12.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:12.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.965+0000 7fdc1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:12.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc20d60640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:12.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc20d60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:12.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc20d60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:12.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:12.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:12.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:12.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc1bfff640 1 --2- 172.21.15.100:0/3002018217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc1c09f6d0 0x7fdc1c121de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc1b7fe640 10 monclient(hunting): handle_auth_done global_id 4397 payload 824 2024-09-13T11:53:12.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc1b7fe640 1 -- 172.21.15.100:0/3002018217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc1c09f6d0 msgr2=0x7fdc1c121de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc1b7fe640 1 --2- 172.21.15.100:0/3002018217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc1c09f6d0 0x7fdc1c121de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc1b7fe640 1 -- 172.21.15.100:0/3002018217 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc1c0a1ec0 msgr2=0x7fdc1c11fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc1b7fe640 1 --2- 172.21.15.100:0/3002018217 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc1c0a1ec0 0x7fdc1c11fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc1b7fe640 10 monclient: _finish_hunting 0 2024-09-13T11:53:12.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc1b7fe640 1 monclient: found mon.noname-c 2024-09-13T11:53:12.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:12.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc1b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:53:12.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc1b7fe640 1 -- 172.21.15.100:0/3002018217 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc1c1235e0 con 0x7fdc1c0a15a0 2024-09-13T11:53:12.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc1bfff640 10 monclient: get_auth_request con 0x7fdc1c09f6d0 auth_method 0 2024-09-13T11:53:12.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc20d60640 1 --2- 172.21.15.100:0/3002018217 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc1c0a1ec0 0x7fdc1c11fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:53:12.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc1bfff640 1 --2- 172.21.15.100:0/3002018217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc1c09f6d0 0x7fdc1c121de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:53:12.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc1b7fe640 1 --2- 172.21.15.100:0/3002018217 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c0a15a0 0x7fdc1c11f8f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdc1000c430 tx=0x7fdc1000c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:12.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc18ff9640 1 -- 172.21.15.100:0/3002018217 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc100074e0 con 0x7fdc1c0a15a0 2024-09-13T11:53:12.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:12.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc18ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:12.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc18ff9640 10 monclient: dump: 2024-09-13T11:53:12.990 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:12.990 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:12.990 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.991 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.991 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:12.991 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.991 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:12.991 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:12.991 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:12.991 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc18ff9640 10 monclient: _finish_auth 0 2024-09-13T11:53:12.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc18ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:53:12.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:12.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc227c3640 5 monclient: authenticate success, global_id 4397 2024-09-13T11:53:12.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc18ff9640 1 -- 172.21.15.100:0/3002018217 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc100047e0 con 0x7fdc1c0a15a0 2024-09-13T11:53:12.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc227c3640 10 monclient: _renew_subs 2024-09-13T11:53:12.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc227c3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:12.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc227c3640 1 -- 172.21.15.100:0/3002018217 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc1c076a90 con 0x7fdc1c0a15a0 2024-09-13T11:53:12.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc18ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:12.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc18ff9640 1 -- 172.21.15.100:0/3002018217 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc10004bb0 con 0x7fdc1c0a15a0 2024-09-13T11:53:12.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:12.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc18ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:12.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc18ff9640 10 monclient: dump: 2024-09-13T11:53:12.992 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:12.992 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:12.992 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.992 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.992 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:12.992 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.993 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:12.993 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:12.993 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:12.993 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc227c3640 10 monclient: _renew_subs 2024-09-13T11:53:12.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc227c3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:12.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc227c3640 1 -- 172.21.15.100:0/3002018217 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc1c123300 con 0x7fdc1c0a15a0 2024-09-13T11:53:12.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.966+0000 7fdc1a7fc640 4 set_mon_vals no callback set 2024-09-13T11:53:12.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.967+0000 7fdc227c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:12.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.967+0000 7fdc227c3640 1 --2- 172.21.15.100:0/3002018217 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c120370 0x7fdc1c11c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.967+0000 7fdc227c3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.967+0000 7fdc227c3640 1 -- 172.21.15.100:0/3002018217 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fdc1c120750 con 0x7fdc1c120370 2024-09-13T11:53:12.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.967+0000 7fdc1bfff640 1 --2- 172.21.15.100:0/3002018217 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c120370 0x7fdc1c11c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.967+0000 7fdc1bfff640 10 monclient: get_auth_request con 0x7fdc1c120370 auth_method 0 2024-09-13T11:53:12.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.967+0000 7fdc1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:12.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.967+0000 7fdc1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:12.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.967+0000 7fdc1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:12.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.967+0000 7fdc18ff9640 1 -- 172.21.15.100:0/3002018217 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fdc10011070 con 0x7fdc1c0a15a0 2024-09-13T11:53:12.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.967+0000 7fdc1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:12.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.967+0000 7fdc1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:12.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.967+0000 7fdc1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:12.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.967+0000 7fdc1bfff640 10 monclient(hunting): handle_auth_done global_id 4400 payload 824 2024-09-13T11:53:12.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.967+0000 7fdc1bfff640 1 -- 172.21.15.100:0/3002018217 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdc1c0a1980 con 0x7fdc1c120370 2024-09-13T11:53:12.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.967+0000 7fdc18ff9640 1 --2- 172.21.15.100:0/3002018217 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fdc0803dfa0 0x7fdc08040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.967+0000 7fdc18ff9640 1 -- 172.21.15.100:0/3002018217 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4374+0+0 (secure 0 0 0) 0x7fdc100521e0 con 0x7fdc1c0a15a0 2024-09-13T11:53:12.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.967+0000 7fdc1bfff640 1 --2- 172.21.15.100:0/3002018217 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c120370 0x7fdc1c11c1e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c010720 tx=0x7fdc0c010be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:12.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.968+0000 7fdc18ff9640 1 -- 172.21.15.100:0/3002018217 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c00f040 con 0x7fdc1c120370 2024-09-13T11:53:12.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.968+0000 7fdc18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:12.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.968+0000 7fdc18ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:12.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.968+0000 7fdc18ff9640 10 monclient: dump: 2024-09-13T11:53:12.995 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:12.995 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:12.995 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.995 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:12.995 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:12.995 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:12.995 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:12.996 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:12.996 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:12.996 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:12.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.968+0000 7fdc20d60640 1 --2- 172.21.15.100:0/3002018217 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fdc0803dfa0 0x7fdc08040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.968+0000 7fdc20d60640 10 monclient: get_auth_request con 0x7fdc0803dfa0 auth_method 0 2024-09-13T11:53:12.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.968+0000 7fdc18ff9640 1 -- 172.21.15.100:0/3002018217 <== mon.2 v2:172.21.15.100:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdc0c01c400 con 0x7fdc1c120370 2024-09-13T11:53:12.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.968+0000 7fdc18ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:12.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.968+0000 7fdc18ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:12.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.968+0000 7fdc18ff9640 1 -- 172.21.15.100:0/3002018217 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c120370 msgr2=0x7fdc1c11c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:12.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.968+0000 7fdc18ff9640 1 --2- 172.21.15.100:0/3002018217 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c120370 0x7fdc1c11c1e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c010720 tx=0x7fdc0c010be0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.968+0000 7fdc18ff9640 1 -- 172.21.15.100:0/3002018217 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c120370 msgr2=0x7fdc1c11c1e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:53:12.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.968+0000 7fdc18ff9640 1 --2- 172.21.15.100:0/3002018217 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c120370 0x7fdc1c11c1e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:12.997 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.968+0000 7fdc20d60640 1 --2- 172.21.15.100:0/3002018217 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fdc0803dfa0 0x7fdc08040460 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c1334a0 tx=0x7fdc1400d000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:12.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.974+0000 7fdc227c3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:53:12.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.974+0000 7fdc227c3640 1 --2- 172.21.15.100:0/3002018217 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c11c720 0x7fdc1c11cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:12.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.974+0000 7fdc227c3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:12.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.974+0000 7fdc227c3640 1 -- 172.21.15.100:0/3002018217 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fdc1c11d070 con 0x7fdc1c11c720 2024-09-13T11:53:12.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.974+0000 7fdc1bfff640 1 --2- 172.21.15.100:0/3002018217 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c11c720 0x7fdc1c11cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:12.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.974+0000 7fdc1bfff640 10 monclient: get_auth_request con 0x7fdc1c11c720 auth_method 0 2024-09-13T11:53:12.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.974+0000 7fdc1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:12.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.974+0000 7fdc1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:12.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.974+0000 7fdc1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:12.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.974+0000 7fdc1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:12.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.974+0000 7fdc1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:12.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.974+0000 7fdc1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:12.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.974+0000 7fdc1bfff640 10 monclient(hunting): handle_auth_done global_id 4403 payload 824 2024-09-13T11:53:12.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.974+0000 7fdc1bfff640 1 -- 172.21.15.100:0/3002018217 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdc1c133f30 con 0x7fdc1c11c720 2024-09-13T11:53:12.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.974+0000 7fdc1bfff640 1 --2- 172.21.15.100:0/3002018217 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c11c720 0x7fdc1c11cb30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c010720 tx=0x7fdc0c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:13.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.974+0000 7fdc18ff9640 1 -- 172.21.15.100:0/3002018217 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c006240 con 0x7fdc1c11c720 2024-09-13T11:53:13.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.974+0000 7fdc18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:13.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.974+0000 7fdc18ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:13.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.974+0000 7fdc18ff9640 10 monclient: dump: 2024-09-13T11:53:13.000 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:13.000 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:13.000 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.000 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.000 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:13.000 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:13.000 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:13.000 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:13.000 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:13.001 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:13.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.975+0000 7fdc18ff9640 1 -- 172.21.15.100:0/3002018217 <== mon.2 v2:172.21.15.100:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7fdc0c006240 con 0x7fdc1c11c720 2024-09-13T11:53:13.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.975+0000 7fdc18ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:53:13.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.975+0000 7fdc18ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:13.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.975+0000 7fdc18ff9640 1 -- 172.21.15.100:0/3002018217 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c11c720 msgr2=0x7fdc1c11cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.975+0000 7fdc18ff9640 1 --2- 172.21.15.100:0/3002018217 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c11c720 0x7fdc1c11cb30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c010720 tx=0x7fdc0c0046c0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.975+0000 7fdc18ff9640 1 -- 172.21.15.100:0/3002018217 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c11c720 msgr2=0x7fdc1c11cb30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:53:13.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.975+0000 7fdc18ff9640 1 --2- 172.21.15.100:0/3002018217 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c11c720 0x7fdc1c11cb30 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c010720 tx=0x7fdc0c0046c0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.001 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:53:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:53:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:53:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "state": "peon", 2024-09-13T11:53:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "election_epoch": 8, 2024-09-13T11:53:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "uptime": 32045, 2024-09-13T11:53:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum": [ 2024-09-13T11:53:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-13T11:53:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: 1, 2024-09-13T11:53:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: 2 2024-09-13T11:53:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_age": 26, 2024-09-13T11:53:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:53:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "required_con": "2450521708007866372", 2024-09-13T11:53:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "required_mon": [ 2024-09-13T11:53:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:53:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:53:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:53:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:53:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:53:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:53:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:53:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:53:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:53:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:53:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:53:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:53:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_mon": [ 2024-09-13T11:53:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:53:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:53:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:53:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:53:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:53:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:53:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:53:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:53:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:53:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:53:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:53:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "outside_quorum": [], 2024-09-13T11:53:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "extra_probe_peers": [], 2024-09-13T11:53:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_provider": [], 2024-09-13T11:53:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": { 2024-09-13T11:53:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-09-13T11:53:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "0b1f4b4d-fad4-4dd3-86ac-a286d80846c0", 2024-09-13T11:53:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:53:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:53:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release": 19, 2024-09-13T11:53:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release_name": "squid", 2024-09-13T11:53:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "election_strategy": 1, 2024-09-13T11:53:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "disallowed_leaders: ": "", 2024-09-13T11:53:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false, 2024-09-13T11:53:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "tiebreaker_mon": "", 2024-09-13T11:53:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_ranks: ": "", 2024-09-13T11:53:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:53:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "persistent": [ 2024-09-13T11:53:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:53:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:53:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:53:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:53:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:53:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:53:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:53:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:53:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:53:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:53:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:53:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "optional": [] 2024-09-13T11:53:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "mons": [ 2024-09-13T11:53:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:53:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:53:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:53:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:53:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:53:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3300", 2024-09-13T11:53:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:53:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789", 2024-09-13T11:53:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789/0", 2024-09-13T11:53:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-13T11:53:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:53:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:53:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:53:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:53:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:53:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:53:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:53:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:53:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:3300", 2024-09-13T11:53:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:53:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789", 2024-09-13T11:53:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789/0", 2024-09-13T11:53:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.122:6789/0", 2024-09-13T11:53:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:53:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:53:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:53:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:53:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:53:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:53:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:53:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:53:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3301", 2024-09-13T11:53:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:53:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790", 2024-09-13T11:53:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790/0", 2024-09-13T11:53:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-13T11:53:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:53:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:53:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:53:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: "feature_map": { 2024-09-13T11:53:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": [ 2024-09-13T11:53:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:53:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:53:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:53:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": [ 2024-09-13T11:53:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:53:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:53:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 2 2024-09-13T11:53:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: "client": [ 2024-09-13T11:53:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:53:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:53:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 3 2024-09-13T11:53:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false 2024-09-13T11:53:13.014 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:53:13.016 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.975+0000 7fdc227c3640 1 -- 172.21.15.100:0/3002018217 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fdc0803dfa0 msgr2=0x7fdc08040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.016 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.975+0000 7fdc227c3640 1 --2- 172.21.15.100:0/3002018217 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fdc0803dfa0 0x7fdc08040460 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c1334a0 tx=0x7fdc1400d000 comp rx=0 tx=0).stop 2024-09-13T11:53:13.016 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.975+0000 7fdc227c3640 10 monclient: shutdown 2024-09-13T11:53:13.016 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.975+0000 7fdc227c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:13.016 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.975+0000 7fdc227c3640 1 -- 172.21.15.100:0/3002018217 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c0a15a0 msgr2=0x7fdc1c11f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.016 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.975+0000 7fdc227c3640 1 --2- 172.21.15.100:0/3002018217 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c0a15a0 0x7fdc1c11f8f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdc1000c430 tx=0x7fdc1000c8f0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.016 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.975+0000 7fdc20d60640 1 -- 172.21.15.100:0/3002018217 reap_dead start 2024-09-13T11:53:13.016 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.975+0000 7fdc227c3640 1 -- 172.21.15.100:0/3002018217 shutdown_connections 2024-09-13T11:53:13.016 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.975+0000 7fdc227c3640 1 --2- 172.21.15.100:0/3002018217 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc1c0a15a0 0x7fdc1c11f8f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.016 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.975+0000 7fdc227c3640 1 -- 172.21.15.100:0/3002018217 >> 172.21.15.100:0/3002018217 conn(0x7fdc1c0a6470 msgr2=0x7fdc1c0a3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:13.016 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.975+0000 7fdc227c3640 1 -- 172.21.15.100:0/3002018217 shutdown_connections 2024-09-13T11:53:13.017 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:12.975+0000 7fdc227c3640 1 -- 172.21.15.100:0/3002018217 wait complete. 2024-09-13T11:53:13.017 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T11:53:13.017 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-13T11:53:13.017 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-13T11:53:13.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.114+0000 7f65216bf640 10 monclient: get_monmap_and_config 2024-09-13T11:53:13.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.114+0000 7f65216bf640 10 monclient: build_initial_monmap 2024-09-13T11:53:13.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.114+0000 7f65216bf640 10 monclient: monmap: 2024-09-13T11:53:13.117 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:13.117 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:13.117 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:13.116143+0000 2024-09-13T11:53:13.117 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:13.116143+0000 2024-09-13T11:53:13.117 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:13.117 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:13.117 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:13.118 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:13.118 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:13.118 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:13.118 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:13.118 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:13.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.114+0000 7f65216bf640 1 Processor -- start 2024-09-13T11:53:13.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.114+0000 7f65216bf640 1 -- start start 2024-09-13T11:53:13.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.114+0000 7f65216bf640 10 monclient: init 2024-09-13T11:53:13.126 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f65216bf640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:13.126 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f65216bf640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-13T11:53:13.126 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f65216bf640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f651c06ea50 0x7f651c0a2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:13.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f65216bf640 10 monclient(hunting): picked mon.noname-b con 0x7f651c06ea50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:13.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f65216bf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f651c06ee30 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:13.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f65216bf640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f651c13c9f0 0x7f651c13ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:13.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f65216bf640 10 monclient(hunting): picked mon.noname-c con 0x7f651c13c9f0 addr v2:172.21.15.122:3300/0 2024-09-13T11:53:13.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f65216bf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f651c0a1590 con 0x7f651c06ee30 2024-09-13T11:53:13.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f65216bf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:13.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f65216bf640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f651c0a1870 con 0x7f651c13c9f0 2024-09-13T11:53:13.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f65216bf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:13.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f65216bf640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f651c0a1b50 con 0x7f651c06ea50 2024-09-13T11:53:13.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f65216bf640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:13.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f65216bf640 10 monclient(hunting): authenticate will time out at 773.709371s 2024-09-13T11:53:13.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651affd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f651c06ea50 0x7f651c0a2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:13.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651affd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f651c06ea50 0x7f651c0a2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:35254/0 (socket says 172.21.15.100:35254) 2024-09-13T11:53:13.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651affd640 1 -- 172.21.15.100:0/1935535978 learned_addr learned my addr 172.21.15.100:0/1935535978 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:13.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651affd640 10 monclient(hunting): get_auth_request con 0x7f651c06ea50 auth_method 0 2024-09-13T11:53:13.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:13.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:13.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:13.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651b7fe640 1 --2- 172.21.15.100:0/1935535978 >> v2:172.21.15.122:3300/0 conn(0x7f651c13c9f0 0x7f651c13ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:13.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:13.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:13.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:13.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651b7fe640 10 monclient(hunting): get_auth_request con 0x7f651c13c9f0 auth_method 0 2024-09-13T11:53:13.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:13.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:13.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:13.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:13.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:13.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:13.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651affd640 10 monclient(hunting): handle_auth_done global_id 4406 payload 274 2024-09-13T11:53:13.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651affd640 1 -- 172.21.15.100:0/1935535978 >> v2:172.21.15.122:3300/0 conn(0x7f651c13c9f0 msgr2=0x7f651c13ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:13.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651affd640 1 --2- 172.21.15.100:0/1935535978 >> v2:172.21.15.122:3300/0 conn(0x7f651c13c9f0 0x7f651c13ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651affd640 1 -- 172.21.15.100:0/1935535978 >> v1:172.21.15.122:6789/0 conn(0x7f651c06ee30 legacy=0x7f651c06f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651affd640 10 monclient: _finish_hunting 0 2024-09-13T11:53:13.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651affd640 1 monclient: found mon.noname-b 2024-09-13T11:53:13.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:13.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:53:13.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.115+0000 7f651affd640 1 -- 172.21.15.100:0/1935535978 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f651c06fca0 con 0x7f651c06ea50 2024-09-13T11:53:13.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.116+0000 7f651affd640 1 --2- 172.21.15.100:0/1935535978 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f651c06ea50 0x7f651c0a2860 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6514009870 tx=0x7f651402a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95c56ec952c587eb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:13.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.116+0000 7f6519ffb640 1 -- 172.21.15.100:0/1935535978 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6514037070 con 0x7f651c06ea50 2024-09-13T11:53:13.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.116+0000 7f6519ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:13.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.116+0000 7f6519ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:13.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.116+0000 7f6519ffb640 10 monclient: dump: 2024-09-13T11:53:13.130 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:13.130 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:13.130 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.130 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.130 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:13.130 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:13.130 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:13.131 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:13.131 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:13.131 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:13.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.116+0000 7f6519ffb640 10 monclient: _finish_auth 0 2024-09-13T11:53:13.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.116+0000 7f6519ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:53:13.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.116+0000 7f6519ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:13.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.116+0000 7f65216bf640 5 monclient: authenticate success, global_id 4406 2024-09-13T11:53:13.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.116+0000 7f65216bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:13.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.116+0000 7f6519ffb640 1 -- 172.21.15.100:0/1935535978 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6514032440 con 0x7f651c06ea50 2024-09-13T11:53:13.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.116+0000 7f6519ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:13.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.116+0000 7f65216bf640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:13.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.116+0000 7f65216bf640 4 set_mon_vals no callback set 2024-09-13T11:53:13.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.116+0000 7f6519ffb640 1 -- 172.21.15.100:0/1935535978 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6514032810 con 0x7f651c06ea50 2024-09-13T11:53:13.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.116+0000 7f6519ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:13.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.116+0000 7f6519ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:13.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.116+0000 7f6519ffb640 10 monclient: dump: 2024-09-13T11:53:13.132 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:13.132 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:13.132 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.132 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.132 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:13.132 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:13.132 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:13.132 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:13.132 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:13.132 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:13.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.117+0000 7f65216bf640 10 monclient: shutdown 2024-09-13T11:53:13.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.117+0000 7f65216bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:13.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.117+0000 7f65216bf640 1 -- 172.21.15.100:0/1935535978 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f651c06ea50 msgr2=0x7f651c0a2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.117+0000 7f65216bf640 1 --2- 172.21.15.100:0/1935535978 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f651c06ea50 0x7f651c0a2860 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6514009870 tx=0x7f651402a130 comp rx=0 tx=0).stop 2024-09-13T11:53:13.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.117+0000 7f65216bf640 1 -- 172.21.15.100:0/1935535978 shutdown_connections 2024-09-13T11:53:13.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.117+0000 7f65216bf640 1 --2- 172.21.15.100:0/1935535978 >> v2:172.21.15.122:3300/0 conn(0x7f651c13c9f0 0x7f651c13ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.117+0000 7f65216bf640 1 --2- 172.21.15.100:0/1935535978 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f651c06ea50 0x7f651c0a2860 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.117+0000 7f65216bf640 1 -- 172.21.15.100:0/1935535978 >> 172.21.15.100:0/1935535978 conn(0x7f651c0a68e0 msgr2=0x7f651c0a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:13.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.117+0000 7f65216bf640 1 -- 172.21.15.100:0/1935535978 shutdown_connections 2024-09-13T11:53:13.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.117+0000 7f65216bf640 1 -- 172.21.15.100:0/1935535978 wait complete. 2024-09-13T11:53:13.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.117+0000 7f65216bf640 10 monclient: build_initial_monmap 2024-09-13T11:53:13.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.117+0000 7f65216bf640 10 monclient: monmap: 2024-09-13T11:53:13.133 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:13.133 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:13.134 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:13.134 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:13.134 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:13.134 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:13.134 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:13.134 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:13.134 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:13.134 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:13.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f65216bf640 1 Processor -- start 2024-09-13T11:53:13.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f65216bf640 1 -- start start 2024-09-13T11:53:13.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f65216bf640 10 monclient: init 2024-09-13T11:53:13.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f65216bf640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:13.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f65216bf640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:53:13.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f65216bf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f651c06ea50 0x7f651c137320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:13.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f65216bf640 10 monclient(hunting): picked mon.noname-b con 0x7f651c06ea50 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:13.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f65216bf640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f651c06ee30 0x7f651c137860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:13.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f65216bf640 10 monclient(hunting): picked mon.noname-a con 0x7f651c06ee30 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:13.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f65216bf640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f651c13c9f0 0x7f651c09ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:13.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f65216bf640 10 monclient(hunting): picked mon.noname-c con 0x7f651c13c9f0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:13.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f65216bf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:13.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f65216bf640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f651c0a5be0 con 0x7f651c06ee30 2024-09-13T11:53:13.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f65216bf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:13.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f65216bf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f651c0a5a60 con 0x7f651c06ea50 2024-09-13T11:53:13.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f65216bf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:13.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f65216bf640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f651c13f320 con 0x7f651c13c9f0 2024-09-13T11:53:13.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f65216bf640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:13.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f65216bf640 10 monclient(hunting): authenticate will time out at 773.712759s 2024-09-13T11:53:13.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f651b7fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f651c13c9f0 0x7f651c09ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:13.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f651b7fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f651c13c9f0 0x7f651c09ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:35268/0 (socket says 172.21.15.100:35268) 2024-09-13T11:53:13.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f651b7fe640 1 -- 172.21.15.100:0/3400000844 learned_addr learned my addr 172.21.15.100:0/3400000844 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:13.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f651b7fe640 10 monclient(hunting): get_auth_request con 0x7f651c13c9f0 auth_method 0 2024-09-13T11:53:13.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f651a7fc640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f651c06ee30 0x7f651c137860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:13.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f651b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:13.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f651b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:13.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f651b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:13.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f651a7fc640 10 monclient(hunting): get_auth_request con 0x7f651c06ee30 auth_method 0 2024-09-13T11:53:13.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f651a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:13.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f651a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:13.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f651a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:13.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f651affd640 1 --2- 172.21.15.100:0/3400000844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f651c06ea50 0x7f651c137320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:13.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f651b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:13.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f651b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:13.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f651b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:13.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f651affd640 10 monclient(hunting): get_auth_request con 0x7f651c06ea50 auth_method 0 2024-09-13T11:53:13.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f651affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:13.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f651affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:13.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.118+0000 7f651affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:13.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f651a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:13.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f651a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:13.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f651a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:13.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f651affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:13.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f651affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:13.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f651affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:13.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f651b7fe640 10 monclient(hunting): handle_auth_done global_id 4409 payload 824 2024-09-13T11:53:13.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f651b7fe640 1 -- 172.21.15.100:0/3400000844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f651c06ea50 msgr2=0x7f651c137320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f651b7fe640 1 --2- 172.21.15.100:0/3400000844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f651c06ea50 0x7f651c137320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f651b7fe640 1 -- 172.21.15.100:0/3400000844 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f651c06ee30 msgr2=0x7f651c137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f651b7fe640 1 --2- 172.21.15.100:0/3400000844 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f651c06ee30 0x7f651c137860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f651b7fe640 10 monclient: _finish_hunting 0 2024-09-13T11:53:13.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f651b7fe640 1 monclient: found mon.noname-c 2024-09-13T11:53:13.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f651b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:13.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f651b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:53:13.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f651b7fe640 1 -- 172.21.15.100:0/3400000844 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f651c121c10 con 0x7f651c13c9f0 2024-09-13T11:53:13.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f651b7fe640 1 --2- 172.21.15.100:0/3400000844 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f651c13c9f0 0x7f651c09ebd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f650c00efc0 tx=0x7f650c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:13.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f65037fe640 1 -- 172.21.15.100:0/3400000844 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c018020 con 0x7f651c13c9f0 2024-09-13T11:53:13.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f65037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:13.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f65037fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:13.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f65037fe640 10 monclient: dump: 2024-09-13T11:53:13.139 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:13.139 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:13.139 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.139 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.139 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:13.139 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:13.139 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:13.139 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:13.139 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:13.139 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:13.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f65037fe640 10 monclient: _finish_auth 0 2024-09-13T11:53:13.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f65037fe640 10 monclient: _check_auth_tickets 2024-09-13T11:53:13.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f65037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:13.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f65037fe640 1 -- 172.21.15.100:0/3400000844 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f650c014480 con 0x7f651c13c9f0 2024-09-13T11:53:13.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f65037fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:13.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f65037fe640 1 -- 172.21.15.100:0/3400000844 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c014870 con 0x7f651c13c9f0 2024-09-13T11:53:13.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f65216bf640 5 monclient: authenticate success, global_id 4409 2024-09-13T11:53:13.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f65216bf640 10 monclient: _renew_subs 2024-09-13T11:53:13.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f65216bf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:13.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f65216bf640 1 -- 172.21.15.100:0/3400000844 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f651c121ef0 con 0x7f651c13c9f0 2024-09-13T11:53:13.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f65037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:13.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f65037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:13.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f65037fe640 10 monclient: dump: 2024-09-13T11:53:13.140 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:13.141 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:13.141 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.141 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.141 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:13.141 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:13.141 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:13.141 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:13.141 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:13.141 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:13.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f65197fa640 4 set_mon_vals no callback set 2024-09-13T11:53:13.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f65216bf640 10 monclient: _renew_subs 2024-09-13T11:53:13.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f65216bf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:13.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.119+0000 7f65216bf640 1 -- 172.21.15.100:0/3400000844 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f651c121930 con 0x7f651c13c9f0 2024-09-13T11:53:13.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.120+0000 7f65216bf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:13.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.120+0000 7f65216bf640 1 --2- 172.21.15.100:0/3400000844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f651c09f8d0 0x7f651c02d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:13.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.120+0000 7f65216bf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:13.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.120+0000 7f65216bf640 1 -- 172.21.15.100:0/3400000844 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f651c0a2860 con 0x7f651c09f8d0 2024-09-13T11:53:13.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.120+0000 7f65037fe640 1 -- 172.21.15.100:0/3400000844 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f650c005d10 con 0x7f651c13c9f0 2024-09-13T11:53:13.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.120+0000 7f651affd640 1 --2- 172.21.15.100:0/3400000844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f651c09f8d0 0x7f651c02d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:13.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.120+0000 7f651affd640 10 monclient: get_auth_request con 0x7f651c09f8d0 auth_method 0 2024-09-13T11:53:13.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.120+0000 7f651affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:13.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.120+0000 7f651affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:13.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.120+0000 7f651affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:13.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.120+0000 7f651affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:13.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.120+0000 7f651affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:13.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.120+0000 7f651affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:13.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.120+0000 7f65037fe640 1 --2- 172.21.15.100:0/3400000844 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f650803dfd0 0x7f6508040490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:13.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.120+0000 7f65037fe640 1 -- 172.21.15.100:0/3400000844 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4374+0+0 (secure 0 0 0) 0x7f650c053140 con 0x7f651c13c9f0 2024-09-13T11:53:13.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.120+0000 7f651affd640 10 monclient(hunting): handle_auth_done global_id 4393 payload 824 2024-09-13T11:53:13.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.120+0000 7f651affd640 1 -- 172.21.15.100:0/3400000844 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f651c13c0f0 con 0x7f651c09f8d0 2024-09-13T11:53:13.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.120+0000 7f651a7fc640 1 --2- 172.21.15.100:0/3400000844 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f650803dfd0 0x7f6508040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:13.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.120+0000 7f651a7fc640 10 monclient: get_auth_request con 0x7f650803dfd0 auth_method 0 2024-09-13T11:53:13.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.120+0000 7f651affd640 1 --2- 172.21.15.100:0/3400000844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f651c09f8d0 0x7f651c02d1f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f651c0a5e80 tx=0x7f65140046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:13.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.121+0000 7f65037fe640 1 -- 172.21.15.100:0/3400000844 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6514043070 con 0x7f651c09f8d0 2024-09-13T11:53:13.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.121+0000 7f65037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:13.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.121+0000 7f65037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:53:13.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.121+0000 7f65037fe640 10 monclient: dump: 2024-09-13T11:53:13.143 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:13.143 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:13.143 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.144 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.144 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:13.144 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:13.144 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:13.144 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:13.144 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:13.144 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:13.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.121+0000 7f651a7fc640 1 --2- 172.21.15.100:0/3400000844 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f650803dfd0 0x7f6508040490 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f651c0702e0 tx=0x7f6510009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:13.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.121+0000 7f65037fe640 1 -- 172.21.15.100:0/3400000844 <== mon.1 v2:172.21.15.122:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f651403c640 con 0x7f651c09f8d0 2024-09-13T11:53:13.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.121+0000 7f65037fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:13.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.121+0000 7f65037fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:13.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.121+0000 7f65037fe640 1 -- 172.21.15.100:0/3400000844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f651c09f8d0 msgr2=0x7f651c02d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.121+0000 7f65037fe640 1 --2- 172.21.15.100:0/3400000844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f651c09f8d0 0x7f651c02d1f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f651c0a5e80 tx=0x7f65140046c0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.145 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.121+0000 7f65037fe640 1 -- 172.21.15.100:0/3400000844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f651c09f8d0 msgr2=0x7f651c02d1f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:53:13.145 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.121+0000 7f65037fe640 1 --2- 172.21.15.100:0/3400000844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f651c09f8d0 0x7f651c02d1f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.127+0000 7f65216bf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:53:13.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.127+0000 7f65216bf640 1 --2- 172.21.15.100:0/3400000844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f651c09fcb0 0x7f651c02d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:13.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.127+0000 7f65216bf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:13.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.127+0000 7f65216bf640 1 -- 172.21.15.100:0/3400000844 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f651c069100 con 0x7f651c09fcb0 2024-09-13T11:53:13.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.127+0000 7f651affd640 1 --2- 172.21.15.100:0/3400000844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f651c09fcb0 0x7f651c02d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:13.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.127+0000 7f651affd640 10 monclient: get_auth_request con 0x7f651c09fcb0 auth_method 0 2024-09-13T11:53:13.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.127+0000 7f651affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:13.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.127+0000 7f651affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:13.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.127+0000 7f651affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:13.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.127+0000 7f651affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:13.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.127+0000 7f651affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:13.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.127+0000 7f651affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:13.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.128+0000 7f651affd640 10 monclient(hunting): handle_auth_done global_id 4396 payload 824 2024-09-13T11:53:13.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.128+0000 7f651affd640 1 -- 172.21.15.100:0/3400000844 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f651c1206a0 con 0x7f651c09fcb0 2024-09-13T11:53:13.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.128+0000 7f651affd640 1 --2- 172.21.15.100:0/3400000844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f651c09fcb0 0x7f651c02d730 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f651c0a29e0 tx=0x7f6514009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:13.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.128+0000 7f65037fe640 1 -- 172.21.15.100:0/3400000844 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6514055020 con 0x7f651c09fcb0 2024-09-13T11:53:13.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.128+0000 7f65037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:13.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.128+0000 7f65037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:53:13.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.128+0000 7f65037fe640 10 monclient: dump: 2024-09-13T11:53:13.150 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:13.150 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:13.150 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.150 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.150 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:13.150 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:13.150 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:13.150 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:13.151 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:13.151 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:13.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.128+0000 7f65037fe640 1 -- 172.21.15.100:0/3400000844 <== mon.1 v2:172.21.15.122:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4886 (secure 0 0 0) 0x7f651403b3a0 con 0x7f651c09fcb0 2024-09-13T11:53:13.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.128+0000 7f65037fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:53:13.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.128+0000 7f65037fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:13.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.128+0000 7f65037fe640 1 -- 172.21.15.100:0/3400000844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f651c09fcb0 msgr2=0x7f651c02d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.128+0000 7f65037fe640 1 --2- 172.21.15.100:0/3400000844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f651c09fcb0 0x7f651c02d730 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f651c0a29e0 tx=0x7f6514009240 comp rx=0 tx=0).stop 2024-09-13T11:53:13.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.128+0000 7f65037fe640 1 -- 172.21.15.100:0/3400000844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f651c09fcb0 msgr2=0x7f651c02d730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:53:13.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.128+0000 7f65037fe640 1 --2- 172.21.15.100:0/3400000844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f651c09fcb0 0x7f651c02d730 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.153 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:53:13.153 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:53:13.153 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:53:13.153 INFO:teuthology.orchestra.run.smithi100.stdout: "state": "peon", 2024-09-13T11:53:13.153 INFO:teuthology.orchestra.run.smithi100.stdout: "election_epoch": 8, 2024-09-13T11:53:13.153 INFO:teuthology.orchestra.run.smithi100.stdout: "uptime": 32159, 2024-09-13T11:53:13.153 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum": [ 2024-09-13T11:53:13.153 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-13T11:53:13.153 INFO:teuthology.orchestra.run.smithi100.stdout: 1, 2024-09-13T11:53:13.153 INFO:teuthology.orchestra.run.smithi100.stdout: 2 2024-09-13T11:53:13.153 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:13.154 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_age": 26, 2024-09-13T11:53:13.154 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:53:13.154 INFO:teuthology.orchestra.run.smithi100.stdout: "required_con": "2450521708007866372", 2024-09-13T11:53:13.154 INFO:teuthology.orchestra.run.smithi100.stdout: "required_mon": [ 2024-09-13T11:53:13.154 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:53:13.154 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:53:13.154 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:53:13.154 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:53:13.154 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:53:13.154 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:53:13.154 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:53:13.154 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:53:13.154 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:53:13.154 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:53:13.155 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:53:13.155 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:13.155 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:53:13.155 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_mon": [ 2024-09-13T11:53:13.155 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:53:13.155 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:53:13.155 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:53:13.155 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:53:13.155 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:53:13.155 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:53:13.155 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:53:13.155 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:53:13.155 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:53:13.156 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:53:13.156 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:53:13.156 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:13.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:13.156 INFO:teuthology.orchestra.run.smithi100.stdout: "outside_quorum": [], 2024-09-13T11:53:13.156 INFO:teuthology.orchestra.run.smithi100.stdout: "extra_probe_peers": [], 2024-09-13T11:53:13.156 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_provider": [], 2024-09-13T11:53:13.156 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": { 2024-09-13T11:53:13.156 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-09-13T11:53:13.156 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "0b1f4b4d-fad4-4dd3-86ac-a286d80846c0", 2024-09-13T11:53:13.156 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:53:13.156 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:53:13.157 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release": 19, 2024-09-13T11:53:13.157 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release_name": "squid", 2024-09-13T11:53:13.157 INFO:teuthology.orchestra.run.smithi100.stdout: "election_strategy": 1, 2024-09-13T11:53:13.157 INFO:teuthology.orchestra.run.smithi100.stdout: "disallowed_leaders: ": "", 2024-09-13T11:53:13.157 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false, 2024-09-13T11:53:13.157 INFO:teuthology.orchestra.run.smithi100.stdout: "tiebreaker_mon": "", 2024-09-13T11:53:13.157 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_ranks: ": "", 2024-09-13T11:53:13.157 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:53:13.157 INFO:teuthology.orchestra.run.smithi100.stdout: "persistent": [ 2024-09-13T11:53:13.157 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:53:13.157 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:53:13.157 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:53:13.157 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:53:13.158 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:53:13.158 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:53:13.158 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:53:13.158 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:53:13.158 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:53:13.158 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:53:13.158 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:53:13.158 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:13.158 INFO:teuthology.orchestra.run.smithi100.stdout: "optional": [] 2024-09-13T11:53:13.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:13.158 INFO:teuthology.orchestra.run.smithi100.stdout: "mons": [ 2024-09-13T11:53:13.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.158 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:53:13.159 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:53:13.159 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:53:13.159 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:53:13.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.159 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:53:13.159 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3300", 2024-09-13T11:53:13.159 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:13.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:13.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.159 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:53:13.159 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789", 2024-09-13T11:53:13.159 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:13.159 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:13.160 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:13.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:13.160 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789/0", 2024-09-13T11:53:13.160 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-13T11:53:13.160 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:53:13.160 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:53:13.160 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:53:13.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:13.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.160 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:53:13.160 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:53:13.160 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:53:13.160 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:53:13.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.161 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:53:13.161 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:3300", 2024-09-13T11:53:13.161 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:13.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:13.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.161 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:53:13.161 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789", 2024-09-13T11:53:13.161 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:13.161 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:13.161 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:13.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:13.161 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789/0", 2024-09-13T11:53:13.162 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.122:6789/0", 2024-09-13T11:53:13.162 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:53:13.162 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:53:13.162 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:53:13.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:13.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.162 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:53:13.162 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:53:13.162 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:53:13.162 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:53:13.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.162 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:53:13.162 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3301", 2024-09-13T11:53:13.163 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:13.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:13.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.163 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:53:13.163 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790", 2024-09-13T11:53:13.163 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:13.163 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:13.163 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:13.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:13.163 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790/0", 2024-09-13T11:53:13.163 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-13T11:53:13.163 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:53:13.163 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:53:13.163 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:53:13.164 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:13.164 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:13.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:13.164 INFO:teuthology.orchestra.run.smithi100.stdout: "feature_map": { 2024-09-13T11:53:13.164 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": [ 2024-09-13T11:53:13.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.164 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:53:13.164 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:53:13.164 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:53:13.164 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:13.164 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:13.164 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": [ 2024-09-13T11:53:13.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.165 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:53:13.165 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:53:13.165 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 3 2024-09-13T11:53:13.165 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:13.165 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:13.165 INFO:teuthology.orchestra.run.smithi100.stdout: "client": [ 2024-09-13T11:53:13.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.165 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:53:13.165 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:53:13.165 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 5 2024-09-13T11:53:13.165 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:13.165 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:13.165 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": [ 2024-09-13T11:53:13.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:13.166 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:53:13.166 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:53:13.166 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:53:13.166 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:13.166 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:13.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:13.166 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false 2024-09-13T11:53:13.166 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:53:13.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.128+0000 7f65216bf640 1 -- 172.21.15.100:0/3400000844 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f650803dfd0 msgr2=0x7f6508040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.128+0000 7f65216bf640 1 --2- 172.21.15.100:0/3400000844 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f650803dfd0 0x7f6508040490 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f651c0702e0 tx=0x7f6510009040 comp rx=0 tx=0).stop 2024-09-13T11:53:13.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.128+0000 7f65216bf640 10 monclient: shutdown 2024-09-13T11:53:13.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.128+0000 7f65216bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:13.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.128+0000 7f65216bf640 1 -- 172.21.15.100:0/3400000844 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f651c13c9f0 msgr2=0x7f651c09ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.128+0000 7f65216bf640 1 --2- 172.21.15.100:0/3400000844 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f651c13c9f0 0x7f651c09ebd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f650c00efc0 tx=0x7f650c00c490 comp rx=0 tx=0).stop 2024-09-13T11:53:13.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.128+0000 7f651b7fe640 1 -- 172.21.15.100:0/3400000844 reap_dead start 2024-09-13T11:53:13.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.128+0000 7f651b7fe640 1 -- 172.21.15.100:0/3400000844 reap_dead start 2024-09-13T11:53:13.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.128+0000 7f65216bf640 1 -- 172.21.15.100:0/3400000844 shutdown_connections 2024-09-13T11:53:13.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.129+0000 7f65216bf640 1 -- 172.21.15.100:0/3400000844 >> 172.21.15.100:0/3400000844 conn(0x7f651c0a68e0 msgr2=0x7f651c13d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:13.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.129+0000 7f65216bf640 1 -- 172.21.15.100:0/3400000844 shutdown_connections 2024-09-13T11:53:13.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.129+0000 7f65216bf640 1 -- 172.21.15.100:0/3400000844 wait complete. 2024-09-13T11:53:13.169 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T11:53:13.169 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-13T11:53:13.169 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-13T11:53:13.169 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-13T11:53:13.170 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a store 2024-09-13T11:53:13.170 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a sync_force --yes-i-really-mean-it 2024-09-13T11:53:13.269 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.266+0000 7f12f30bf640 10 monclient: get_monmap_and_config 2024-09-13T11:53:13.269 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.266+0000 7f12f30bf640 10 monclient: build_initial_monmap 2024-09-13T11:53:13.269 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.266+0000 7f12f30bf640 10 monclient: monmap: 2024-09-13T11:53:13.269 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:13.269 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:13.269 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:13.268429+0000 2024-09-13T11:53:13.269 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:13.268429+0000 2024-09-13T11:53:13.269 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:13.269 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:13.270 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:13.270 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:13.270 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:13.270 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:13.270 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:13.270 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:13.279 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.267+0000 7f12f30bf640 1 Processor -- start 2024-09-13T11:53:13.280 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.267+0000 7f12f30bf640 1 -- start start 2024-09-13T11:53:13.280 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.267+0000 7f12f30bf640 10 monclient: init 2024-09-13T11:53:13.280 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.267+0000 7f12f30bf640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:13.280 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.267+0000 7f12f30bf640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-13T11:53:13.281 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.267+0000 7f12f30bf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f12ec0a3650 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:13.281 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.267+0000 7f12f30bf640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f12ec0a4140 0x7f12ec13a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:13.281 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.267+0000 7f12f30bf640 10 monclient(hunting): picked mon.noname-a con 0x7f12ec0a4140 addr v2:172.21.15.100:3300/0 2024-09-13T11:53:13.281 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.267+0000 7f12f30bf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12ec13ad90 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:13.281 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.267+0000 7f12f30bf640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12ec0a0d20 con 0x7f12ec0a3650 2024-09-13T11:53:13.281 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.267+0000 7f12f30bf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12ec0a1000 con 0x7f12ec13ad90 2024-09-13T11:53:13.281 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.267+0000 7f12f30bf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:13.281 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.267+0000 7f12f30bf640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f12ec070f40 con 0x7f12ec0a4140 2024-09-13T11:53:13.281 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.267+0000 7f12f30bf640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:13.281 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.267+0000 7f12f30bf640 10 monclient(hunting): authenticate will time out at 773.861886s 2024-09-13T11:53:13.281 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.267+0000 7f12ebfff640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f12ec0a4140 0x7f12ec13a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:13.281 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.267+0000 7f12f0e5b640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f12ec0a3650 0x7f12ec0a3a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:34180/0 (socket says 172.21.15.100:34180) 2024-09-13T11:53:13.281 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.268+0000 7f12f0e5b640 1 -- 172.21.15.100:0/3690882566 learned_addr learned my addr 172.21.15.100:0/3690882566 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:53:13.282 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.267+0000 7f12ebfff640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f12ec0a4140 0x7f12ec13a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:36508/0 (socket says 172.21.15.100:36508) 2024-09-13T11:53:13.282 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.268+0000 7f12ebfff640 10 monclient(hunting): get_auth_request con 0x7f12ec0a4140 auth_method 0 2024-09-13T11:53:13.282 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.268+0000 7f12ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:13.282 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.268+0000 7f12ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:13.283 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.268+0000 7f12ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:13.284 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.268+0000 7f12ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:13.284 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.268+0000 7f12ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:13.284 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.268+0000 7f12ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:13.284 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.268+0000 7f12ebfff640 10 monclient(hunting): handle_auth_done global_id 4458 payload 274 2024-09-13T11:53:13.284 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.268+0000 7f12ebfff640 1 -- 172.21.15.100:0/3690882566 >> v1:172.21.15.122:6789/0 conn(0x7f12ec13ad90 legacy=0x7f12ec13d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.285 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.268+0000 7f12ebfff640 1 -- 172.21.15.100:0/3690882566 >> v1:172.21.15.100:6789/0 conn(0x7f12ec0a3650 legacy=0x7f12ec0a3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.285 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.268+0000 7f12ebfff640 10 monclient: _finish_hunting 0 2024-09-13T11:53:13.285 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.268+0000 7f12ebfff640 1 monclient: found mon.noname-a 2024-09-13T11:53:13.285 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.268+0000 7f12ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:13.285 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.268+0000 7f12ebfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:13.285 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.268+0000 7f12ebfff640 1 -- 172.21.15.100:0/3690882566 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12ec09f9d0 con 0x7f12ec0a4140 2024-09-13T11:53:13.285 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.268+0000 7f12eb7fe640 1 -- 172.21.15.100:0/3690882566 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2283254412 0 0) 0x7f12ec0a0d20 con 0x7f12ec0a3650 2024-09-13T11:53:13.285 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.268+0000 7f12eb7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:53:13.286 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.268+0000 7f12ebfff640 1 --2- 172.21.15.100:0/3690882566 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec0a4140 0x7f12ec13a850 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f12d80097a0 tx=0x7f12d802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cfb284e263a940b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:13.286 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12eb7fe640 1 -- 172.21.15.100:0/3690882566 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d8037070 con 0x7f12ec0a4140 2024-09-13T11:53:13.286 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:13.286 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12eb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:13.286 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12eb7fe640 10 monclient: dump: 2024-09-13T11:53:13.286 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:13.286 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:13.286 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.286 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.286 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:13.286 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:13.287 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:13.287 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:13.287 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:13.287 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:13.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12eb7fe640 10 monclient: _finish_auth 0 2024-09-13T11:53:13.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12eb7fe640 10 monclient: _check_auth_tickets 2024-09-13T11:53:13.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:13.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12f30bf640 5 monclient: authenticate success, global_id 4458 2024-09-13T11:53:13.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12f30bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:13.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12eb7fe640 1 -- 172.21.15.100:0/3690882566 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12d8032440 con 0x7f12ec0a4140 2024-09-13T11:53:13.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12eb7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:13.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12eb7fe640 1 -- 172.21.15.100:0/3690882566 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d802ae60 con 0x7f12ec0a4140 2024-09-13T11:53:13.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12f30bf640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:13.288 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12f30bf640 4 set_mon_vals no callback set 2024-09-13T11:53:13.288 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:13.288 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:13.288 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12eb7fe640 10 monclient: dump: 2024-09-13T11:53:13.288 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:13.288 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:13.288 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.288 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.288 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:13.288 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:13.288 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:13.288 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:13.288 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:13.289 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:13.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12f30bf640 10 monclient: shutdown 2024-09-13T11:53:13.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12f30bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:13.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12f30bf640 1 -- 172.21.15.100:0/3690882566 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec0a4140 msgr2=0x7f12ec13a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12f30bf640 1 --2- 172.21.15.100:0/3690882566 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec0a4140 0x7f12ec13a850 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f12d80097a0 tx=0x7f12d802a110 comp rx=0 tx=0).stop 2024-09-13T11:53:13.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12f30bf640 1 -- 172.21.15.100:0/3690882566 shutdown_connections 2024-09-13T11:53:13.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12f30bf640 1 --2- 172.21.15.100:0/3690882566 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec0a4140 0x7f12ec13a850 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12f30bf640 1 -- 172.21.15.100:0/3690882566 >> 172.21.15.100:0/3690882566 conn(0x7f12ec0a8270 msgr2=0x7f12ec0a4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:13.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12f30bf640 1 -- 172.21.15.100:0/3690882566 shutdown_connections 2024-09-13T11:53:13.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.269+0000 7f12f30bf640 1 -- 172.21.15.100:0/3690882566 wait complete. 2024-09-13T11:53:13.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.270+0000 7f12f30bf640 10 monclient: build_initial_monmap 2024-09-13T11:53:13.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.270+0000 7f12f30bf640 10 monclient: monmap: 2024-09-13T11:53:13.289 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:13.290 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:13.290 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:13.290 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:13.290 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:13.290 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:13.290 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:13.290 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:13.290 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:13.290 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:13.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.270+0000 7f12f30bf640 1 Processor -- start 2024-09-13T11:53:13.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.270+0000 7f12f30bf640 1 -- start start 2024-09-13T11:53:13.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.270+0000 7f12f30bf640 10 monclient: init 2024-09-13T11:53:13.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f30bf640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:13.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f30bf640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:53:13.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f30bf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12ec0a3650 0x7f12ec131cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:13.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f30bf640 10 monclient(hunting): picked mon.noname-b con 0x7f12ec0a3650 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:13.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f30bf640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec0a4140 0x7f12ec06e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:13.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f30bf640 10 monclient(hunting): picked mon.noname-a con 0x7f12ec0a4140 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:13.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f30bf640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12ec13ad90 0x7f12ec06ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:13.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f30bf640 10 monclient(hunting): picked mon.noname-c con 0x7f12ec13ad90 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:13.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f30bf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:13.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f30bf640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec0a1ff0 con 0x7f12ec0a4140 2024-09-13T11:53:13.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f30bf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:13.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f30bf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec0a1310 con 0x7f12ec0a3650 2024-09-13T11:53:13.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f30bf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:13.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f30bf640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f12ec136ea0 con 0x7f12ec13ad90 2024-09-13T11:53:13.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f30bf640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:13.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f30bf640 10 monclient(hunting): authenticate will time out at 773.865366s 2024-09-13T11:53:13.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12ebfff640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec0a4140 0x7f12ec06e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:13.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12ebfff640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec0a4140 0x7f12ec06e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:36516/0 (socket says 172.21.15.100:36516) 2024-09-13T11:53:13.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12ebfff640 1 -- 172.21.15.100:0/2528068726 learned_addr learned my addr 172.21.15.100:0/2528068726 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:13.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12ebfff640 10 monclient(hunting): get_auth_request con 0x7f12ec0a4140 auth_method 0 2024-09-13T11:53:13.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:13.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:13.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:13.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f165c640 1 --2- 172.21.15.100:0/2528068726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12ec13ad90 0x7f12ec06ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:13.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f165c640 10 monclient(hunting): get_auth_request con 0x7f12ec13ad90 auth_method 0 2024-09-13T11:53:13.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f165c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:13.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f165c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:13.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f165c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:13.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:13.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:13.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:13.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f165c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:13.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f165c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:13.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f165c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:13.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f0e5b640 1 --2- 172.21.15.100:0/2528068726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12ec0a3650 0x7f12ec131cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:13.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12ebfff640 10 monclient(hunting): handle_auth_done global_id 4464 payload 824 2024-09-13T11:53:13.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12ebfff640 1 -- 172.21.15.100:0/2528068726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12ec13ad90 msgr2=0x7f12ec06ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12ebfff640 1 --2- 172.21.15.100:0/2528068726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12ec13ad90 0x7f12ec06ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12ebfff640 1 -- 172.21.15.100:0/2528068726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12ec0a3650 msgr2=0x7f12ec131cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12ebfff640 1 --2- 172.21.15.100:0/2528068726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12ec0a3650 0x7f12ec131cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12ebfff640 10 monclient: _finish_hunting 0 2024-09-13T11:53:13.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12ebfff640 1 monclient: found mon.noname-a 2024-09-13T11:53:13.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:13.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12ebfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:53:13.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12ebfff640 1 -- 172.21.15.100:0/2528068726 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12ec076a90 con 0x7f12ec0a4140 2024-09-13T11:53:13.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f0e5b640 10 monclient: get_auth_request con 0x7f12ec0a3650 auth_method 0 2024-09-13T11:53:13.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12f0e5b640 1 --2- 172.21.15.100:0/2528068726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12ec0a3650 0x7f12ec131cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:53:13.295 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.271+0000 7f12ebfff640 1 --2- 172.21.15.100:0/2528068726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec0a4140 0x7f12ec06e950 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f12d80097a0 tx=0x7f12d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:13.295 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.272+0000 7f12e97fa640 1 -- 172.21.15.100:0/2528068726 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d8037050 con 0x7f12ec0a4140 2024-09-13T11:53:13.295 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.272+0000 7f12e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:13.295 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.272+0000 7f12e97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:53:13.295 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.272+0000 7f12e97fa640 10 monclient: dump: 2024-09-13T11:53:13.295 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:13.295 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:13.295 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.295 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.295 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:13.295 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:13.295 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:13.295 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:13.295 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:13.296 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:13.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.272+0000 7f12e97fa640 10 monclient: _finish_auth 0 2024-09-13T11:53:13.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.272+0000 7f12e97fa640 10 monclient: _check_auth_tickets 2024-09-13T11:53:13.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.272+0000 7f12e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:13.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.272+0000 7f12e97fa640 1 -- 172.21.15.100:0/2528068726 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12d8038040 con 0x7f12ec0a4140 2024-09-13T11:53:13.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.272+0000 7f12f30bf640 5 monclient: authenticate success, global_id 4464 2024-09-13T11:53:13.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.272+0000 7f12f30bf640 10 monclient: _renew_subs 2024-09-13T11:53:13.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.272+0000 7f12f30bf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:13.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.272+0000 7f12f30bf640 1 -- 172.21.15.100:0/2528068726 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12ec1383c0 con 0x7f12ec0a4140 2024-09-13T11:53:13.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.272+0000 7f12e97fa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:13.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.272+0000 7f12e97fa640 1 -- 172.21.15.100:0/2528068726 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d8004470 con 0x7f12ec0a4140 2024-09-13T11:53:13.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.272+0000 7f12e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:13.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.272+0000 7f12e97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:13.297 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.272+0000 7f12e97fa640 10 monclient: dump: 2024-09-13T11:53:13.297 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:13.297 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:13.297 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.297 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.297 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:13.297 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:13.297 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:13.297 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:13.297 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:13.297 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:13.297 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.272+0000 7f12f30bf640 10 monclient: _renew_subs 2024-09-13T11:53:13.297 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.272+0000 7f12eaffd640 4 set_mon_vals no callback set 2024-09-13T11:53:13.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.272+0000 7f12f30bf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:53:13.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.272+0000 7f12f30bf640 1 -- 172.21.15.100:0/2528068726 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12ec070940 con 0x7f12ec0a4140 2024-09-13T11:53:13.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.272+0000 7f12f30bf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:13.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.272+0000 7f12f30bf640 1 --2- 172.21.15.100:0/2528068726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec06f3d0 0x7f12ec0710d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:13.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12f30bf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:13.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12f30bf640 1 -- 172.21.15.100:0/2528068726 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec0a2170 con 0x7f12ec06f3d0 2024-09-13T11:53:13.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12e97fa640 1 -- 172.21.15.100:0/2528068726 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f12d8030900 con 0x7f12ec0a4140 2024-09-13T11:53:13.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12f0e5b640 1 --2- 172.21.15.100:0/2528068726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec06f3d0 0x7f12ec0710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:13.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12f0e5b640 10 monclient: get_auth_request con 0x7f12ec06f3d0 auth_method 0 2024-09-13T11:53:13.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12f0e5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:13.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12f0e5b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:13.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12f0e5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:13.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12f0e5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:13.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12f0e5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:13.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12f0e5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:13.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12e97fa640 1 --2- 172.21.15.100:0/2528068726 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f12d003dfa0 0x7f12d0040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:13.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12e97fa640 1 -- 172.21.15.100:0/2528068726 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4374+0+0 (secure 0 0 0) 0x7f12d80731d0 con 0x7f12ec0a4140 2024-09-13T11:53:13.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12f0e5b640 10 monclient(hunting): handle_auth_done global_id 4467 payload 824 2024-09-13T11:53:13.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12f0e5b640 1 -- 172.21.15.100:0/2528068726 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f12ec06f7b0 con 0x7f12ec06f3d0 2024-09-13T11:53:13.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12f165c640 1 --2- 172.21.15.100:0/2528068726 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f12d003dfa0 0x7f12d0040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:13.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12f165c640 10 monclient: get_auth_request con 0x7f12d003dfa0 auth_method 0 2024-09-13T11:53:13.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12f0e5b640 1 --2- 172.21.15.100:0/2528068726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec06f3d0 0x7f12ec0710d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f12e0000c00 tx=0x7f12e000d950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:13.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12e97fa640 1 -- 172.21.15.100:0/2528068726 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e0018070 con 0x7f12ec06f3d0 2024-09-13T11:53:13.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:13.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12e97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:13.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12e97fa640 10 monclient: dump: 2024-09-13T11:53:13.300 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:13.300 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:13.300 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.300 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.300 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:13.300 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:13.300 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:13.300 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:13.300 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:13.300 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:13.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12e97fa640 1 -- 172.21.15.100:0/2528068726 <== mon.0 v2:172.21.15.100:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f12e00075a0 con 0x7f12ec06f3d0 2024-09-13T11:53:13.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12f165c640 1 --2- 172.21.15.100:0/2528068726 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f12d003dfa0 0x7f12d0040460 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f12e40012e0 tx=0x7f12e4009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:13.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12e97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:13.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12e97fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:13.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12e97fa640 1 -- 172.21.15.100:0/2528068726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec06f3d0 msgr2=0x7f12ec0710d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12e97fa640 1 --2- 172.21.15.100:0/2528068726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec06f3d0 0x7f12ec0710d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f12e0000c00 tx=0x7f12e000d950 comp rx=0 tx=0).stop 2024-09-13T11:53:13.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12e97fa640 1 -- 172.21.15.100:0/2528068726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec06f3d0 msgr2=0x7f12ec0710d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:53:13.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.273+0000 7f12e97fa640 1 --2- 172.21.15.100:0/2528068726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec06f3d0 0x7f12ec0710d0 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f12e0000c00 tx=0x7f12e000d950 comp rx=0 tx=0).stop 2024-09-13T11:53:13.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.282+0000 7f12f30bf640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-13T11:53:13.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.282+0000 7f12f30bf640 1 --2- 172.21.15.100:0/2528068726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec071610 0x7f12ec0719f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:13.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.282+0000 7f12f30bf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:13.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.282+0000 7f12f30bf640 1 -- 172.21.15.100:0/2528068726 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec06ca60 con 0x7f12ec071610 2024-09-13T11:53:13.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.282+0000 7f12f0e5b640 1 --2- 172.21.15.100:0/2528068726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec071610 0x7f12ec0719f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:13.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.282+0000 7f12f0e5b640 10 monclient: get_auth_request con 0x7f12ec071610 auth_method 0 2024-09-13T11:53:13.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.282+0000 7f12f0e5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:13.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.282+0000 7f12f0e5b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:13.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.282+0000 7f12f0e5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:13.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.282+0000 7f12f0e5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:13.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.282+0000 7f12f0e5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:13.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.282+0000 7f12f0e5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:13.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.283+0000 7f12f0e5b640 10 monclient(hunting): handle_auth_done global_id 4470 payload 824 2024-09-13T11:53:13.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.283+0000 7f12f0e5b640 1 -- 172.21.15.100:0/2528068726 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f12ec13a550 con 0x7f12ec071610 2024-09-13T11:53:13.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.283+0000 7f12f0e5b640 1 --2- 172.21.15.100:0/2528068726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec071610 0x7f12ec0719f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f12e0015000 tx=0x7f12e0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:13.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.285+0000 7f12e97fa640 1 -- 172.21.15.100:0/2528068726 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e0018050 con 0x7f12ec071610 2024-09-13T11:53:13.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.285+0000 7f12e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:13.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.285+0000 7f12e97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:53:13.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.285+0000 7f12e97fa640 10 monclient: dump: 2024-09-13T11:53:13.305 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:13.305 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:13.305 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.305 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.305 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:13.305 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:13.305 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:13.305 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:13.305 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:13.305 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:13.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.286+0000 7f12e97fa640 1 -- 172.21.15.100:0/2528068726 <== mon.0 v2:172.21.15.100:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f12e0019040 con 0x7f12ec071610 2024-09-13T11:53:13.306 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.286+0000 7f12e97fa640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-13T11:53:13.306 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.286+0000 7f12e97fa640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:13.306 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.286+0000 7f12e97fa640 1 -- 172.21.15.100:0/2528068726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec071610 msgr2=0x7f12ec0719f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.306 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.286+0000 7f12e97fa640 1 --2- 172.21.15.100:0/2528068726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec071610 0x7f12ec0719f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f12e0015000 tx=0x7f12e0000ed0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.306 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.286+0000 7f12e97fa640 1 -- 172.21.15.100:0/2528068726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec071610 msgr2=0x7f12ec0719f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:53:13.306 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.286+0000 7f12e97fa640 1 --2- 172.21.15.100:0/2528068726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec071610 0x7f12ec0719f0 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f12e0015000 tx=0x7f12e0000ed0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.306 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:53:13.306 INFO:teuthology.orchestra.run.smithi100.stdout: "ret": 0, 2024-09-13T11:53:13.306 INFO:teuthology.orchestra.run.smithi100.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-13T11:53:13.306 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:53:13.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.287+0000 7f12f30bf640 1 -- 172.21.15.100:0/2528068726 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f12d003dfa0 msgr2=0x7f12d0040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.287+0000 7f12f30bf640 1 --2- 172.21.15.100:0/2528068726 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f12d003dfa0 0x7f12d0040460 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f12e40012e0 tx=0x7f12e4009040 comp rx=0 tx=0).stop 2024-09-13T11:53:13.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.287+0000 7f12f30bf640 10 monclient: shutdown 2024-09-13T11:53:13.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.287+0000 7f12f30bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:13.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.287+0000 7f12f30bf640 1 -- 172.21.15.100:0/2528068726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec0a4140 msgr2=0x7f12ec06e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.287+0000 7f12f30bf640 1 --2- 172.21.15.100:0/2528068726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec0a4140 0x7f12ec06e950 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f12d80097a0 tx=0x7f12d80046c0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.287+0000 7f12f165c640 1 -- 172.21.15.100:0/2528068726 reap_dead start 2024-09-13T11:53:13.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.287+0000 7f12f30bf640 1 -- 172.21.15.100:0/2528068726 shutdown_connections 2024-09-13T11:53:13.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.287+0000 7f12f30bf640 1 --2- 172.21.15.100:0/2528068726 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12ec0a4140 0x7f12ec06e950 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.287+0000 7f12f30bf640 1 -- 172.21.15.100:0/2528068726 >> 172.21.15.100:0/2528068726 conn(0x7f12ec0a8270 msgr2=0x7f12ec135ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:13.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.287+0000 7f12f30bf640 1 -- 172.21.15.100:0/2528068726 shutdown_connections 2024-09-13T11:53:13.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.288+0000 7f12f30bf640 1 -- 172.21.15.100:0/2528068726 wait complete. 2024-09-13T11:53:13.309 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-13T11:53:13.309 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-13T11:53:13.309 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:53:13.320 INFO:tasks.ceph.mon.a:Stopped 2024-09-13T11:53:13.320 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-13T11:53:13.320 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:53:13.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.454+0000 7f5e6e8e4640 10 monclient: get_monmap_and_config 2024-09-13T11:53:13.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.454+0000 7f5e6e8e4640 10 monclient: build_initial_monmap 2024-09-13T11:53:13.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.454+0000 7f5e6e8e4640 10 monclient: monmap: 2024-09-13T11:53:13.457 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:13.457 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:13.457 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:13.456101+0000 2024-09-13T11:53:13.457 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:13.456101+0000 2024-09-13T11:53:13.457 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:13.458 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:13.458 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:13.458 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:13.458 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:13.459 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:13.459 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:13.459 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:13.467 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.454+0000 7f5e6e8e4640 1 Processor -- start 2024-09-13T11:53:13.467 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.455+0000 7f5e6e8e4640 1 -- start start 2024-09-13T11:53:13.467 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.455+0000 7f5e6e8e4640 10 monclient: init 2024-09-13T11:53:13.467 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.455+0000 7f5e6e8e4640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:13.467 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.455+0000 7f5e6e8e4640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-13T11:53:13.467 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.455+0000 7f5e6e8e4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5e68137870 0x7f5e6809ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:13.467 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.455+0000 7f5e6e8e4640 10 monclient(hunting): picked mon.noname-c con 0x7f5e68137870 addr v2:172.21.15.122:3300/0 2024-09-13T11:53:13.467 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.455+0000 7f5e6e8e4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e68137c50 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:13.467 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.455+0000 7f5e6e8e4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e680a2d50 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:13.467 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.455+0000 7f5e6e8e4640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e680a44e0 con 0x7f5e68137c50 2024-09-13T11:53:13.467 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.455+0000 7f5e6e8e4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e68076cc0 con 0x7f5e680a2d50 2024-09-13T11:53:13.468 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.455+0000 7f5e6e8e4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:13.468 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.455+0000 7f5e6e8e4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f5e6809c6c0 con 0x7f5e68137870 2024-09-13T11:53:13.468 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.455+0000 7f5e6e8e4640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:13.468 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.455+0000 7f5e6e8e4640 10 monclient(hunting): authenticate will time out at 774.049567s 2024-09-13T11:53:13.468 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.455+0000 7f5e677fe640 1 -- >> v1:172.21.15.100:6789/0 conn(0x7f5e68137c50 legacy=0x7f5e680a2640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6789/0 2024-09-13T11:53:13.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.455+0000 7f5e67fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5e68137870 0x7f5e6809ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:13.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.455+0000 7f5e67fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5e68137870 0x7f5e6809ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.100:36130/0 (socket says 172.21.15.100:36130) 2024-09-13T11:53:13.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.455+0000 7f5e6ce81640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f5e680a2d50 0x7f5e6813f680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.100:56482/0 (socket says 172.21.15.100:56482) 2024-09-13T11:53:13.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.455+0000 7f5e67fff640 1 -- 172.21.15.100:0/946289724 learned_addr learned my addr 172.21.15.100:0/946289724 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:13.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.455+0000 7f5e67fff640 10 monclient(hunting): get_auth_request con 0x7f5e68137870 auth_method 0 2024-09-13T11:53:13.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.455+0000 7f5e67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:13.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.455+0000 7f5e67fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:13.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.455+0000 7f5e67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:13.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.456+0000 7f5e67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:13.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.456+0000 7f5e67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:13.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.456+0000 7f5e67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:13.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.456+0000 7f5e67fff640 10 monclient(hunting): handle_auth_done global_id 4402 payload 274 2024-09-13T11:53:13.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.456+0000 7f5e67fff640 1 -- 172.21.15.100:0/946289724 >> v1:172.21.15.122:6789/0 conn(0x7f5e680a2d50 legacy=0x7f5e6813f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.456+0000 7f5e67fff640 1 -- 172.21.15.100:0/946289724 >> v1:172.21.15.100:6789/0 conn(0x7f5e68137c50 legacy=0x7f5e680a2640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:53:13.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.456+0000 7f5e67fff640 10 monclient: _finish_hunting 0 2024-09-13T11:53:13.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.456+0000 7f5e67fff640 1 monclient: found mon.noname-c 2024-09-13T11:53:13.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.456+0000 7f5e67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:13.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.456+0000 7f5e67fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.122:3300/0 2024-09-13T11:53:13.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.456+0000 7f5e67fff640 1 -- 172.21.15.100:0/946289724 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e6809bd90 con 0x7f5e68137870 2024-09-13T11:53:13.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e67fff640 1 --2- 172.21.15.100:0/946289724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e68137870 0x7f5e6809ca50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5e58009870 tx=0x7f5e5802a130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a6b508a8d3c8449 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:13.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e66ffd640 1 -- 172.21.15.100:0/946289724 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e58037070 con 0x7f5e68137870 2024-09-13T11:53:13.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:13.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e66ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:13.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e66ffd640 10 monclient: dump: 2024-09-13T11:53:13.470 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:13.470 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:13.470 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.471 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.471 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:13.471 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:13.471 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:13.471 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:13.471 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:13.471 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:13.471 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e66ffd640 10 monclient: _finish_auth 0 2024-09-13T11:53:13.471 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e66ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:53:13.471 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:13.471 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e66ffd640 1 -- 172.21.15.100:0/946289724 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e58032440 con 0x7f5e68137870 2024-09-13T11:53:13.471 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e6e8e4640 5 monclient: authenticate success, global_id 4402 2024-09-13T11:53:13.471 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e6e8e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:13.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e66ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:13.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e6e8e4640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:13.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e66ffd640 1 -- 172.21.15.100:0/946289724 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e58032810 con 0x7f5e68137870 2024-09-13T11:53:13.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:13.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e6e8e4640 4 set_mon_vals no callback set 2024-09-13T11:53:13.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e66ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:53:13.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e66ffd640 10 monclient: dump: 2024-09-13T11:53:13.472 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:13.472 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:13.472 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.472 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.472 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:13.472 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:13.473 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:13.473 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:13.473 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:13.473 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:13.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e6e8e4640 10 monclient: shutdown 2024-09-13T11:53:13.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e6e8e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:13.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e6e8e4640 1 -- 172.21.15.100:0/946289724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e68137870 msgr2=0x7f5e6809ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e6e8e4640 1 --2- 172.21.15.100:0/946289724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e68137870 0x7f5e6809ca50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5e58009870 tx=0x7f5e5802a130 comp rx=0 tx=0).stop 2024-09-13T11:53:13.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e6e8e4640 1 -- 172.21.15.100:0/946289724 shutdown_connections 2024-09-13T11:53:13.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e6e8e4640 1 --2- 172.21.15.100:0/946289724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e68137870 0x7f5e6809ca50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e6e8e4640 1 -- 172.21.15.100:0/946289724 >> 172.21.15.100:0/946289724 conn(0x7f5e680a7260 msgr2=0x7f5e680a3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:13.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e6e8e4640 1 -- 172.21.15.100:0/946289724 shutdown_connections 2024-09-13T11:53:13.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.457+0000 7f5e6e8e4640 1 -- 172.21.15.100:0/946289724 wait complete. 2024-09-13T11:53:13.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.458+0000 7f5e6e8e4640 10 monclient: build_initial_monmap 2024-09-13T11:53:13.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.458+0000 7f5e6e8e4640 10 monclient: monmap: 2024-09-13T11:53:13.474 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:13.474 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:13.474 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:13.474 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:13.474 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:13.474 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:13.474 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:13.474 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:13.474 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:13.474 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:13.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.458+0000 7f5e6e8e4640 1 Processor -- start 2024-09-13T11:53:13.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.458+0000 7f5e6e8e4640 1 -- start start 2024-09-13T11:53:13.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.458+0000 7f5e6e8e4640 10 monclient: init 2024-09-13T11:53:13.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e6e8e4640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:13.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e6e8e4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:53:13.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e6e8e4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e680a2d50 0x7f5e681364a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:13.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e6e8e4640 10 monclient(hunting): picked mon.noname-b con 0x7f5e680a2d50 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:13.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e6e8e4640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e68137870 0x7f5e681369e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:13.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e6e8e4640 10 monclient(hunting): picked mon.noname-c con 0x7f5e68137870 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:13.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e6e8e4640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e68137c50 0x7f5e68120040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:13.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e6e8e4640 10 monclient(hunting): picked mon.noname-a con 0x7f5e68137c50 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:13.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e6e8e4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:13.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e6e8e4640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f5e68074690 con 0x7f5e68137c50 2024-09-13T11:53:13.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e6e8e4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:13.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e6e8e4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f5e68074510 con 0x7f5e680a2d50 2024-09-13T11:53:13.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e6e8e4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:13.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e6e8e4640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f5e68138080 con 0x7f5e68137870 2024-09-13T11:53:13.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e6e8e4640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:13.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e6e8e4640 10 monclient(hunting): authenticate will time out at 774.053357s 2024-09-13T11:53:13.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e6ce81640 1 -- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e68137c50 msgr2=0x7f5e68120040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:3300/0 2024-09-13T11:53:13.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e6ce81640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e68137c50 0x7f5e68120040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:53:13.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e677fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e68137870 0x7f5e681369e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:13.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e677fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e68137870 0x7f5e681369e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:35308/0 (socket says 172.21.15.100:35308) 2024-09-13T11:53:13.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e677fe640 1 -- 172.21.15.100:0/2227506931 learned_addr learned my addr 172.21.15.100:0/2227506931 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:13.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e677fe640 10 monclient(hunting): get_auth_request con 0x7f5e68137870 auth_method 0 2024-09-13T11:53:13.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:13.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e677fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:13.477 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:13.477 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:13.477 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:13.477 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:13.477 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e67fff640 1 --2- 172.21.15.100:0/2227506931 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e680a2d50 0x7f5e681364a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:13.477 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e677fe640 10 monclient(hunting): handle_auth_done global_id 4415 payload 824 2024-09-13T11:53:13.477 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e677fe640 1 -- 172.21.15.100:0/2227506931 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e680a2d50 msgr2=0x7f5e681364a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.477 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e677fe640 1 --2- 172.21.15.100:0/2227506931 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e680a2d50 0x7f5e681364a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.477 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e677fe640 1 -- 172.21.15.100:0/2227506931 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e68137c50 msgr2=0x7f5e68120040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:53:13.477 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e677fe640 1 --2- 172.21.15.100:0/2227506931 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e68137c50 0x7f5e68120040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.477 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e677fe640 10 monclient: _finish_hunting 0 2024-09-13T11:53:13.477 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e677fe640 1 monclient: found mon.noname-c 2024-09-13T11:53:13.477 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:13.477 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e677fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:53:13.478 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e677fe640 1 -- 172.21.15.100:0/2227506931 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e681cd3d0 con 0x7f5e68137870 2024-09-13T11:53:13.478 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e67fff640 10 monclient: get_auth_request con 0x7f5e680a2d50 auth_method 0 2024-09-13T11:53:13.478 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e67fff640 1 --2- 172.21.15.100:0/2227506931 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e680a2d50 0x7f5e681364a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:53:13.478 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.459+0000 7f5e677fe640 1 --2- 172.21.15.100:0/2227506931 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e68137870 0x7f5e681369e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5e5000b550 tx=0x7f5e5000ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:13.478 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.460+0000 7f5e64ff9640 1 -- 172.21.15.100:0/2227506931 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e50013020 con 0x7f5e68137870 2024-09-13T11:53:13.478 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.460+0000 7f5e64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:13.478 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.460+0000 7f5e64ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:13.478 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.460+0000 7f5e64ff9640 10 monclient: dump: 2024-09-13T11:53:13.478 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:13.478 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:13.478 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.478 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.479 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:13.479 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:13.479 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:13.479 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:13.479 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:13.479 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:13.479 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.460+0000 7f5e64ff9640 10 monclient: _finish_auth 0 2024-09-13T11:53:13.479 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.460+0000 7f5e64ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:53:13.479 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.460+0000 7f5e64ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:13.479 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.460+0000 7f5e64ff9640 1 -- 172.21.15.100:0/2227506931 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e50003de0 con 0x7f5e68137870 2024-09-13T11:53:13.479 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.460+0000 7f5e6e8e4640 5 monclient: authenticate success, global_id 4415 2024-09-13T11:53:13.479 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.460+0000 7f5e6e8e4640 10 monclient: _renew_subs 2024-09-13T11:53:13.479 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.460+0000 7f5e6e8e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:13.480 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.460+0000 7f5e6e8e4640 1 -- 172.21.15.100:0/2227506931 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e680a0e50 con 0x7f5e68137870 2024-09-13T11:53:13.480 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.460+0000 7f5e64ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:13.480 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.460+0000 7f5e64ff9640 1 -- 172.21.15.100:0/2227506931 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e500052b0 con 0x7f5e68137870 2024-09-13T11:53:13.480 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.460+0000 7f5e64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:13.480 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.460+0000 7f5e64ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:13.480 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.460+0000 7f5e64ff9640 10 monclient: dump: 2024-09-13T11:53:13.480 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:13.480 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:13.480 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.480 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:13.480 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:13.480 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:13.480 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:13.481 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:13.481 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:13.481 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:13.481 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.460+0000 7f5e667fc640 4 set_mon_vals no callback set 2024-09-13T11:53:13.481 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.460+0000 7f5e6e8e4640 10 monclient: _renew_subs 2024-09-13T11:53:13.481 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.460+0000 7f5e6e8e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:13.481 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.460+0000 7f5e6e8e4640 1 -- 172.21.15.100:0/2227506931 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e68123b80 con 0x7f5e68137870 2024-09-13T11:53:13.481 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.461+0000 7f5e64ff9640 1 -- 172.21.15.100:0/2227506931 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f5e50016020 con 0x7f5e68137870 2024-09-13T11:53:13.481 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.461+0000 7f5e6e8e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:13.481 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.461+0000 7f5e6e8e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:13.481 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.461+0000 7f5e6e8e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:13.481 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.461+0000 7f5e6e8e4640 1 -- 172.21.15.100:0/2227506931 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e680a3170 con 0x7f5e68137870 2024-09-13T11:53:13.481 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.461+0000 7f5e64ff9640 1 --2- 172.21.15.100:0/2227506931 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f5e4803df50 0x7f5e48040410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:13.481 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.461+0000 7f5e64ff9640 1 -- 172.21.15.100:0/2227506931 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4738+0+0 (secure 0 0 0) 0x7f5e50052480 con 0x7f5e68137870 2024-09-13T11:53:13.482 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.461+0000 7f5e67fff640 1 --2- 172.21.15.100:0/2227506931 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f5e4803df50 0x7f5e48040410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:13.482 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.461+0000 7f5e67fff640 10 monclient: get_auth_request con 0x7f5e4803df50 auth_method 0 2024-09-13T11:53:13.482 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.462+0000 7f5e67fff640 1 --2- 172.21.15.100:0/2227506931 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f5e4803df50 0x7f5e48040410 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5e58034000 tx=0x7f5e5802e000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:13.482 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.465+0000 7f5e64ff9640 1 -- 172.21.15.100:0/2227506931 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f5e5001d020 con 0x7f5e68137870 2024-09-13T11:53:13.482 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.465+0000 7f5e64ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:13.482 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.465+0000 7f5e64ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:13.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.705+0000 7f5e6e8e4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:53:13.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.705+0000 7f5e6e8e4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:53:13.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.705+0000 7f5e6e8e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:13.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.705+0000 7f5e6e8e4640 1 -- 172.21.15.100:0/2227506931 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5e6806a080 con 0x7f5e68137870 2024-09-13T11:53:13.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.705+0000 7f5e64ff9640 1 -- 172.21.15.100:0/2227506931 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f5e5000e3c0 con 0x7f5e68137870 2024-09-13T11:53:13.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.705+0000 7f5e64ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:53:13.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.705+0000 7f5e64ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:13.708 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":27,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:53:13.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.707+0000 7f5e6e8e4640 1 -- 172.21.15.100:0/2227506931 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f5e4803df50 msgr2=0x7f5e48040410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.707+0000 7f5e6e8e4640 1 --2- 172.21.15.100:0/2227506931 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f5e4803df50 0x7f5e48040410 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5e58034000 tx=0x7f5e5802e000 comp rx=0 tx=0).stop 2024-09-13T11:53:13.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.707+0000 7f5e6e8e4640 10 monclient: shutdown 2024-09-13T11:53:13.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.707+0000 7f5e6e8e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:13.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.707+0000 7f5e6e8e4640 1 -- 172.21.15.100:0/2227506931 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e68137870 msgr2=0x7f5e681369e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:13.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.707+0000 7f5e6e8e4640 1 --2- 172.21.15.100:0/2227506931 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e68137870 0x7f5e681369e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5e5000b550 tx=0x7f5e5000ba10 comp rx=0 tx=0).stop 2024-09-13T11:53:13.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.707+0000 7f5e6e8e4640 1 -- 172.21.15.100:0/2227506931 shutdown_connections 2024-09-13T11:53:13.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.707+0000 7f5e6e8e4640 1 --2- 172.21.15.100:0/2227506931 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f5e4803df50 0x7f5e48040410 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.707+0000 7f5e6e8e4640 1 --2- 172.21.15.100:0/2227506931 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e68137c50 0x7f5e68120040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.707+0000 7f5e6e8e4640 1 --2- 172.21.15.100:0/2227506931 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e68137870 0x7f5e681369e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.707+0000 7f5e6e8e4640 1 --2- 172.21.15.100:0/2227506931 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e680a2d50 0x7f5e681364a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:13.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.707+0000 7f5e6e8e4640 1 -- 172.21.15.100:0/2227506931 >> 172.21.15.100:0/2227506931 conn(0x7f5e680a7260 msgr2=0x7f5e680a6100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:13.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.707+0000 7f5e6e8e4640 1 -- 172.21.15.100:0/2227506931 shutdown_connections 2024-09-13T11:53:13.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:13.707+0000 7f5e6e8e4640 1 -- 172.21.15.100:0/2227506931 wait complete. 2024-09-13T11:53:16.721 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:53:16.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.828+0000 7f36bac50640 10 monclient: get_monmap_and_config 2024-09-13T11:53:16.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.828+0000 7f36bac50640 10 monclient: build_initial_monmap 2024-09-13T11:53:16.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.828+0000 7f36bac50640 10 monclient: monmap: 2024-09-13T11:53:16.830 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:16.830 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:16.830 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:16.829304+0000 2024-09-13T11:53:16.830 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:16.829304+0000 2024-09-13T11:53:16.830 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:16.830 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:16.830 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:16.830 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:16.831 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:16.831 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:16.831 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:16.831 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:16.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.829+0000 7f36bac50640 1 Processor -- start 2024-09-13T11:53:16.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.829+0000 7f36bac50640 1 -- start start 2024-09-13T11:53:16.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.829+0000 7f36bac50640 10 monclient: init 2024-09-13T11:53:16.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.829+0000 7f36bac50640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:16.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.829+0000 7f36bac50640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-13T11:53:16.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.829+0000 7f36bac50640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b40a37d0 0x7f36b40a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:16.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.829+0000 7f36bac50640 10 monclient(hunting): picked mon.noname-b con 0x7f36b40a37d0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:16.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.829+0000 7f36bac50640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f36b406eeb0 0x7f36b40a40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:16.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.829+0000 7f36bac50640 10 monclient(hunting): picked mon.noname-a con 0x7f36b406eeb0 addr v2:172.21.15.100:3300/0 2024-09-13T11:53:16.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.829+0000 7f36bac50640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f36b4137970 0x7f36b4137e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:16.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.829+0000 7f36bac50640 10 monclient(hunting): picked mon.noname-c con 0x7f36b4137970 addr v2:172.21.15.122:3300/0 2024-09-13T11:53:16.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.829+0000 7f36bac50640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:16.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.829+0000 7f36bac50640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f36b409fde0 con 0x7f36b406eeb0 2024-09-13T11:53:16.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.829+0000 7f36bac50640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:16.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.829+0000 7f36bac50640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f36b406f330 con 0x7f36b4137970 2024-09-13T11:53:16.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.829+0000 7f36bac50640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:16.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.829+0000 7f36bac50640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f36b40a4630 con 0x7f36b40a37d0 2024-09-13T11:53:16.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.829+0000 7f36bac50640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:16.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.829+0000 7f36bac50640 10 monclient(hunting): authenticate will time out at 777.422963s 2024-09-13T11:53:16.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.829+0000 7f36b3fff640 1 -- >> v2:172.21.15.100:3300/0 conn(0x7f36b406eeb0 msgr2=0x7f36b40a40f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.100:3300/0 2024-09-13T11:53:16.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.830+0000 7f36b3fff640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f36b406eeb0 0x7f36b40a40f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:53:16.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.830+0000 7f36b89ec640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b40a37d0 0x7f36b40a3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:16.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.830+0000 7f36b89ec640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b40a37d0 0x7f36b40a3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:35322/0 (socket says 172.21.15.100:35322) 2024-09-13T11:53:16.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.830+0000 7f36b89ec640 1 -- 172.21.15.100:0/117581381 learned_addr learned my addr 172.21.15.100:0/117581381 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:16.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.830+0000 7f36b89ec640 10 monclient(hunting): get_auth_request con 0x7f36b40a37d0 auth_method 0 2024-09-13T11:53:16.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.830+0000 7f36b89ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:16.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.830+0000 7f36b89ec640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:16.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.830+0000 7f36b89ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:16.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.830+0000 7f36b89ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:16.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.830+0000 7f36b89ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:16.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.830+0000 7f36b89ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:16.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.830+0000 7f36b91ed640 1 --2- 172.21.15.100:0/117581381 >> v2:172.21.15.122:3300/0 conn(0x7f36b4137970 0x7f36b4137e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:16.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.830+0000 7f36b89ec640 10 monclient(hunting): handle_auth_done global_id 4418 payload 274 2024-09-13T11:53:16.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.830+0000 7f36b89ec640 1 -- 172.21.15.100:0/117581381 >> v2:172.21.15.122:3300/0 conn(0x7f36b4137970 msgr2=0x7f36b4137e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:16.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.830+0000 7f36b89ec640 1 --2- 172.21.15.100:0/117581381 >> v2:172.21.15.122:3300/0 conn(0x7f36b4137970 0x7f36b4137e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:16.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.830+0000 7f36b89ec640 1 -- 172.21.15.100:0/117581381 >> v2:172.21.15.100:3300/0 conn(0x7f36b406eeb0 msgr2=0x7f36b40a40f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:53:16.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.830+0000 7f36b89ec640 1 --2- 172.21.15.100:0/117581381 >> v2:172.21.15.100:3300/0 conn(0x7f36b406eeb0 0x7f36b40a40f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:16.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.830+0000 7f36b89ec640 10 monclient: _finish_hunting 0 2024-09-13T11:53:16.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.830+0000 7f36b89ec640 1 monclient: found mon.noname-b 2024-09-13T11:53:16.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.830+0000 7f36b89ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:16.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.830+0000 7f36b89ec640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:53:16.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.830+0000 7f36b89ec640 1 -- 172.21.15.100:0/117581381 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36b406e950 con 0x7f36b40a37d0 2024-09-13T11:53:16.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36b89ec640 1 --2- 172.21.15.100:0/117581381 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b40a37d0 0x7f36b40a3bb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f36b406ed70 tx=0x7f36a802a0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c61b3a1d32a83a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:16.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36b37fe640 1 -- 172.21.15.100:0/117581381 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a8037070 con 0x7f36b40a37d0 2024-09-13T11:53:16.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:16.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36b37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:16.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36b37fe640 10 monclient: dump: 2024-09-13T11:53:16.844 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:16.844 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:16.844 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:16.845 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:16.845 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:16.845 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:16.845 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:16.845 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:16.845 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:16.845 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:16.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36b37fe640 10 monclient: _finish_auth 0 2024-09-13T11:53:16.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36b37fe640 10 monclient: _check_auth_tickets 2024-09-13T11:53:16.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:16.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36b37fe640 1 -- 172.21.15.100:0/117581381 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36a802ae20 con 0x7f36b40a37d0 2024-09-13T11:53:16.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36bac50640 5 monclient: authenticate success, global_id 4418 2024-09-13T11:53:16.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36bac50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:16.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36b37fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:16.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36bac50640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:16.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36bac50640 4 set_mon_vals no callback set 2024-09-13T11:53:16.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36b37fe640 1 -- 172.21.15.100:0/117581381 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a8032660 con 0x7f36b40a37d0 2024-09-13T11:53:16.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:16.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36b37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:16.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36b37fe640 10 monclient: dump: 2024-09-13T11:53:16.846 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:16.846 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:16.846 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:16.847 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:16.847 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:16.847 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:16.847 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:16.847 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:16.847 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:16.847 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:16.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36bac50640 10 monclient: shutdown 2024-09-13T11:53:16.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36bac50640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:16.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36bac50640 1 -- 172.21.15.100:0/117581381 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b40a37d0 msgr2=0x7f36b40a3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:16.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36bac50640 1 --2- 172.21.15.100:0/117581381 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b40a37d0 0x7f36b40a3bb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f36b406ed70 tx=0x7f36a802a0d0 comp rx=0 tx=0).stop 2024-09-13T11:53:16.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36bac50640 1 -- 172.21.15.100:0/117581381 shutdown_connections 2024-09-13T11:53:16.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36bac50640 1 --2- 172.21.15.100:0/117581381 >> v2:172.21.15.122:3300/0 conn(0x7f36b4137970 0x7f36b4137e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:16.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36bac50640 1 --2- 172.21.15.100:0/117581381 >> v2:172.21.15.100:3300/0 conn(0x7f36b406eeb0 0x7f36b40a40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:16.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36bac50640 1 --2- 172.21.15.100:0/117581381 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b40a37d0 0x7f36b40a3bb0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:16.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.831+0000 7f36bac50640 1 -- 172.21.15.100:0/117581381 >> 172.21.15.100:0/117581381 conn(0x7f36b40a83f0 msgr2=0x7f36b40a48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:16.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.832+0000 7f36bac50640 1 -- 172.21.15.100:0/117581381 shutdown_connections 2024-09-13T11:53:16.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.832+0000 7f36bac50640 1 -- 172.21.15.100:0/117581381 wait complete. 2024-09-13T11:53:16.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.832+0000 7f36bac50640 10 monclient: build_initial_monmap 2024-09-13T11:53:16.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.832+0000 7f36bac50640 10 monclient: monmap: 2024-09-13T11:53:16.848 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:16.848 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:16.848 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:16.849 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:16.849 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:16.849 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:16.849 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:16.849 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:16.849 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:16.849 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:16.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.832+0000 7f36bac50640 1 Processor -- start 2024-09-13T11:53:16.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.832+0000 7f36bac50640 1 -- start start 2024-09-13T11:53:16.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.832+0000 7f36bac50640 10 monclient: init 2024-09-13T11:53:16.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36bac50640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:16.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36bac50640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:53:16.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36bac50640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b406eeb0 0x7f36b4124120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:16.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36bac50640 10 monclient(hunting): picked mon.noname-c con 0x7f36b406eeb0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:16.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36bac50640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36b40a37d0 0x7f36b4124660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:16.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36bac50640 10 monclient(hunting): picked mon.noname-b con 0x7f36b40a37d0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:16.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36bac50640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36b4137970 0x7f36b4124ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:16.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36bac50640 10 monclient(hunting): picked mon.noname-a con 0x7f36b4137970 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:16.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36bac50640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:16.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36bac50640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f36b409edd0 con 0x7f36b4137970 2024-09-13T11:53:16.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36bac50640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:16.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36bac50640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f36b409ec50 con 0x7f36b40a37d0 2024-09-13T11:53:16.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36bac50640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:16.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36bac50640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f36b4138360 con 0x7f36b406eeb0 2024-09-13T11:53:16.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36bac50640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:16.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36bac50640 10 monclient(hunting): authenticate will time out at 777.426602s 2024-09-13T11:53:16.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b91ed640 1 -- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36b4137970 msgr2=0x7f36b4124ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:3300/0 2024-09-13T11:53:16.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b91ed640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36b4137970 0x7f36b4124ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:53:16.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b89ec640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b406eeb0 0x7f36b4124120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:16.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b89ec640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b406eeb0 0x7f36b4124120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:35330/0 (socket says 172.21.15.100:35330) 2024-09-13T11:53:16.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b89ec640 1 -- 172.21.15.100:0/16806195 learned_addr learned my addr 172.21.15.100:0/16806195 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:16.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b89ec640 10 monclient(hunting): get_auth_request con 0x7f36b406eeb0 auth_method 0 2024-09-13T11:53:16.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b89ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:16.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b89ec640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:16.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b89ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:16.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b89ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:16.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b89ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:16.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b3fff640 1 --2- 172.21.15.100:0/16806195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36b40a37d0 0x7f36b4124660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:16.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b89ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:16.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b3fff640 10 monclient(hunting): get_auth_request con 0x7f36b40a37d0 auth_method 0 2024-09-13T11:53:16.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:16.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:16.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:16.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:16.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:16.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:16.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b89ec640 10 monclient(hunting): handle_auth_done global_id 4421 payload 824 2024-09-13T11:53:16.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b89ec640 1 -- 172.21.15.100:0/16806195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36b40a37d0 msgr2=0x7f36b4124660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:16.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b89ec640 1 --2- 172.21.15.100:0/16806195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36b40a37d0 0x7f36b4124660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:16.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b89ec640 1 -- 172.21.15.100:0/16806195 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36b4137970 msgr2=0x7f36b4124ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:53:16.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b89ec640 1 --2- 172.21.15.100:0/16806195 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36b4137970 0x7f36b4124ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:16.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b89ec640 10 monclient: _finish_hunting 0 2024-09-13T11:53:16.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b89ec640 1 monclient: found mon.noname-c 2024-09-13T11:53:16.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b89ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:16.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b89ec640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:53:16.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.833+0000 7f36b89ec640 1 -- 172.21.15.100:0/16806195 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36b409f0b0 con 0x7f36b406eeb0 2024-09-13T11:53:16.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.834+0000 7f36b89ec640 1 --2- 172.21.15.100:0/16806195 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b406eeb0 0x7f36b4124120 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f36a802a110 tx=0x7f36a80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:16.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.834+0000 7f36b17fa640 1 -- 172.21.15.100:0/16806195 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a8041070 con 0x7f36b406eeb0 2024-09-13T11:53:16.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.834+0000 7f36b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:16.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.834+0000 7f36b17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:16.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.834+0000 7f36b17fa640 10 monclient: dump: 2024-09-13T11:53:16.855 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:16.855 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:16.855 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:16.855 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:16.855 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:16.855 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:16.855 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:16.855 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:16.855 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:16.855 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:16.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.834+0000 7f36b17fa640 10 monclient: _finish_auth 0 2024-09-13T11:53:16.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.834+0000 7f36b17fa640 10 monclient: _check_auth_tickets 2024-09-13T11:53:16.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.834+0000 7f36b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:16.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.834+0000 7f36b17fa640 1 -- 172.21.15.100:0/16806195 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36a803c670 con 0x7f36b406eeb0 2024-09-13T11:53:16.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.834+0000 7f36bac50640 5 monclient: authenticate success, global_id 4421 2024-09-13T11:53:16.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.834+0000 7f36bac50640 10 monclient: _renew_subs 2024-09-13T11:53:16.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.834+0000 7f36bac50640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:16.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.834+0000 7f36bac50640 1 -- 172.21.15.100:0/16806195 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36b4127980 con 0x7f36b406eeb0 2024-09-13T11:53:16.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.834+0000 7f36b17fa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:16.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.834+0000 7f36b17fa640 1 -- 172.21.15.100:0/16806195 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a8037040 con 0x7f36b406eeb0 2024-09-13T11:53:16.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.834+0000 7f36b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:16.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.834+0000 7f36b17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:16.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.834+0000 7f36b17fa640 10 monclient: dump: 2024-09-13T11:53:16.857 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:16.857 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:16.857 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:16.857 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:16.857 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:16.857 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:16.857 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:16.857 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:16.857 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:16.857 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:16.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.834+0000 7f36b2ffd640 4 set_mon_vals no callback set 2024-09-13T11:53:16.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.834+0000 7f36bac50640 10 monclient: _renew_subs 2024-09-13T11:53:16.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.834+0000 7f36bac50640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:16.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.834+0000 7f36bac50640 1 -- 172.21.15.100:0/16806195 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f36b40a75b0 con 0x7f36b406eeb0 2024-09-13T11:53:16.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.835+0000 7f36b17fa640 1 -- 172.21.15.100:0/16806195 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f36a8044070 con 0x7f36b406eeb0 2024-09-13T11:53:16.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.835+0000 7f36bac50640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:16.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.835+0000 7f36bac50640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:16.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.835+0000 7f36bac50640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:16.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.835+0000 7f36bac50640 1 -- 172.21.15.100:0/16806195 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36b41270d0 con 0x7f36b406eeb0 2024-09-13T11:53:16.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.835+0000 7f36b17fa640 1 --2- 172.21.15.100:0/16806195 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f369803df80 0x7f3698040440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:16.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.835+0000 7f36b17fa640 1 -- 172.21.15.100:0/16806195 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4738+0+0 (secure 0 0 0) 0x7f36a8072e70 con 0x7f36b406eeb0 2024-09-13T11:53:16.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.835+0000 7f36b3fff640 1 --2- 172.21.15.100:0/16806195 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f369803df80 0x7f3698040440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:16.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.835+0000 7f36b3fff640 10 monclient: get_auth_request con 0x7f369803df80 auth_method 0 2024-09-13T11:53:16.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.836+0000 7f36b3fff640 1 --2- 172.21.15.100:0/16806195 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f369803df80 0x7f3698040440 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f36b4127e70 tx=0x7f36a0009110 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:16.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.840+0000 7f36b17fa640 1 -- 172.21.15.100:0/16806195 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f36b409f0b0 con 0x7f36b406eeb0 2024-09-13T11:53:16.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.840+0000 7f36b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:16.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:16.840+0000 7f36b17fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:17.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:17.081+0000 7f36bac50640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:53:17.083 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:17.081+0000 7f36bac50640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:53:17.083 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:17.081+0000 7f36bac50640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:17.083 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:17.081+0000 7f36bac50640 1 -- 172.21.15.100:0/16806195 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f36b4137e20 con 0x7f36b406eeb0 2024-09-13T11:53:17.083 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:17.081+0000 7f36b17fa640 1 -- 172.21.15.100:0/16806195 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f36a803d3f0 con 0x7f36b406eeb0 2024-09-13T11:53:17.083 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:17.081+0000 7f36b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:53:17.083 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:17.081+0000 7f36b17fa640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:17.083 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":30,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:53:17.085 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:17.083+0000 7f36bac50640 1 -- 172.21.15.100:0/16806195 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f369803df80 msgr2=0x7f3698040440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:17.085 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:17.083+0000 7f36bac50640 1 --2- 172.21.15.100:0/16806195 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f369803df80 0x7f3698040440 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f36b4127e70 tx=0x7f36a0009110 comp rx=0 tx=0).stop 2024-09-13T11:53:17.085 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:17.083+0000 7f36bac50640 10 monclient: shutdown 2024-09-13T11:53:17.085 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:17.083+0000 7f36bac50640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:17.085 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:17.083+0000 7f36bac50640 1 -- 172.21.15.100:0/16806195 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b406eeb0 msgr2=0x7f36b4124120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:17.085 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:17.083+0000 7f36bac50640 1 --2- 172.21.15.100:0/16806195 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b406eeb0 0x7f36b4124120 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f36a802a110 tx=0x7f36a80046c0 comp rx=0 tx=0).stop 2024-09-13T11:53:17.085 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:17.083+0000 7f36bac50640 1 -- 172.21.15.100:0/16806195 shutdown_connections 2024-09-13T11:53:17.085 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:17.083+0000 7f36bac50640 1 --2- 172.21.15.100:0/16806195 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f369803df80 0x7f3698040440 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:17.085 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:17.083+0000 7f36bac50640 1 --2- 172.21.15.100:0/16806195 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36b4137970 0x7f36b4124ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:17.085 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:17.083+0000 7f36bac50640 1 --2- 172.21.15.100:0/16806195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36b40a37d0 0x7f36b4124660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:17.086 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:17.083+0000 7f36bac50640 1 --2- 172.21.15.100:0/16806195 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b406eeb0 0x7f36b4124120 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:17.086 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:17.083+0000 7f36bac50640 1 -- 172.21.15.100:0/16806195 >> 172.21.15.100:0/16806195 conn(0x7f36b40a83f0 msgr2=0x7f36b40a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:17.086 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:17.083+0000 7f36bac50640 1 -- 172.21.15.100:0/16806195 shutdown_connections 2024-09-13T11:53:17.086 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:17.083+0000 7f36bac50640 1 -- 172.21.15.100:0/16806195 wait complete. 2024-09-13T11:53:20.096 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:53:20.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.203+0000 7f9cf331f640 10 monclient: get_monmap_and_config 2024-09-13T11:53:20.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.203+0000 7f9cf331f640 10 monclient: build_initial_monmap 2024-09-13T11:53:20.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.203+0000 7f9cf331f640 10 monclient: monmap: 2024-09-13T11:53:20.205 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:20.205 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:20.205 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:20.204694+0000 2024-09-13T11:53:20.205 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:20.204694+0000 2024-09-13T11:53:20.205 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:20.206 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:20.206 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:20.206 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:20.206 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:20.206 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:20.206 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:20.206 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:20.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.204+0000 7f9cf331f640 1 Processor -- start 2024-09-13T11:53:20.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.204+0000 7f9cf331f640 1 -- start start 2024-09-13T11:53:20.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.204+0000 7f9cf331f640 10 monclient: init 2024-09-13T11:53:20.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf331f640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:20.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf331f640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-13T11:53:20.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf331f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9cec0a37d0 0x7f9cec0a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:20.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf331f640 10 monclient(hunting): picked mon.noname-c con 0x7f9cec0a37d0 addr v2:172.21.15.122:3300/0 2024-09-13T11:53:20.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf331f640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cec06eeb0 0x7f9cec0a40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:20.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf331f640 10 monclient(hunting): picked mon.noname-b con 0x7f9cec06eeb0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:20.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf331f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9cec137970 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:20.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf331f640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cec06e950 con 0x7f9cec137970 2024-09-13T11:53:20.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf331f640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:20.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf331f640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f9cec09fde0 con 0x7f9cec0a37d0 2024-09-13T11:53:20.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf331f640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:20.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf331f640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f9cec06f330 con 0x7f9cec06eeb0 2024-09-13T11:53:20.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf331f640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:20.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf331f640 10 monclient(hunting): authenticate will time out at 780.798229s 2024-09-13T11:53:20.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf18bc640 1 -- >> v1:172.21.15.100:6789/0 conn(0x7f9cec137970 legacy=0x7f9cec137e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6789/0 2024-09-13T11:53:20.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf08ba640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cec06eeb0 0x7f9cec0a40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:20.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf08ba640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cec06eeb0 0x7f9cec0a40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:35340/0 (socket says 172.21.15.100:35340) 2024-09-13T11:53:20.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf08ba640 1 -- 172.21.15.100:0/2499444836 learned_addr learned my addr 172.21.15.100:0/2499444836 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:20.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf08ba640 10 monclient(hunting): get_auth_request con 0x7f9cec06eeb0 auth_method 0 2024-09-13T11:53:20.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf08ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:20.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf08ba640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:20.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf08ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:20.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf10bb640 1 --2- 172.21.15.100:0/2499444836 >> v2:172.21.15.122:3300/0 conn(0x7f9cec0a37d0 0x7f9cec0a3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:20.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf10bb640 10 monclient(hunting): get_auth_request con 0x7f9cec0a37d0 auth_method 0 2024-09-13T11:53:20.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf10bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:20.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf10bb640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:20.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf10bb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:20.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf08ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:20.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf08ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:20.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf08ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:20.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf10bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:20.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf10bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:20.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf10bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:20.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf08ba640 10 monclient(hunting): handle_auth_done global_id 4424 payload 274 2024-09-13T11:53:20.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf08ba640 1 -- 172.21.15.100:0/2499444836 >> v2:172.21.15.122:3300/0 conn(0x7f9cec0a37d0 msgr2=0x7f9cec0a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:20.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf08ba640 1 --2- 172.21.15.100:0/2499444836 >> v2:172.21.15.122:3300/0 conn(0x7f9cec0a37d0 0x7f9cec0a3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:20.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf08ba640 1 -- 172.21.15.100:0/2499444836 >> v1:172.21.15.100:6789/0 conn(0x7f9cec137970 legacy=0x7f9cec137e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:53:20.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf08ba640 10 monclient: _finish_hunting 0 2024-09-13T11:53:20.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf08ba640 1 monclient: found mon.noname-b 2024-09-13T11:53:20.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf08ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:20.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf08ba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:53:20.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.205+0000 7f9cf08ba640 1 -- 172.21.15.100:0/2499444836 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cec0a5240 con 0x7f9cec06eeb0 2024-09-13T11:53:20.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.206+0000 7f9cf08ba640 1 --2- 172.21.15.100:0/2499444836 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cec06eeb0 0x7f9cec0a40f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc009870 tx=0x7f9cdc02a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2fa5fdc3b6ffdf5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:20.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.206+0000 7f9ce3fff640 1 -- 172.21.15.100:0/2499444836 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc037070 con 0x7f9cec06eeb0 2024-09-13T11:53:20.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.206+0000 7f9ce3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:20.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.206+0000 7f9ce3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:20.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.206+0000 7f9ce3fff640 10 monclient: dump: 2024-09-13T11:53:20.219 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:20.219 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:20.219 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:20.219 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:20.219 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:20.219 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:20.220 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:20.220 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:20.220 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:20.220 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:20.220 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.206+0000 7f9ce3fff640 10 monclient: _finish_auth 0 2024-09-13T11:53:20.220 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.206+0000 7f9ce3fff640 10 monclient: _check_auth_tickets 2024-09-13T11:53:20.220 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.206+0000 7f9ce3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:20.220 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.206+0000 7f9ce3fff640 1 -- 172.21.15.100:0/2499444836 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cdc02adf0 con 0x7f9cec06eeb0 2024-09-13T11:53:20.220 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.206+0000 7f9cf331f640 5 monclient: authenticate success, global_id 4424 2024-09-13T11:53:20.220 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.206+0000 7f9cf331f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:20.220 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.206+0000 7f9ce3fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:20.220 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.206+0000 7f9cf331f640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:20.220 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.206+0000 7f9cf331f640 4 set_mon_vals no callback set 2024-09-13T11:53:20.221 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.206+0000 7f9ce3fff640 1 -- 172.21.15.100:0/2499444836 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc032660 con 0x7f9cec06eeb0 2024-09-13T11:53:20.221 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.206+0000 7f9ce3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:20.221 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.206+0000 7f9ce3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:20.221 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.206+0000 7f9ce3fff640 10 monclient: dump: 2024-09-13T11:53:20.221 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:20.221 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:20.221 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:20.221 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:20.221 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:20.221 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:20.221 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:20.221 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:20.221 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:20.222 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:20.222 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.207+0000 7f9cf331f640 10 monclient: shutdown 2024-09-13T11:53:20.222 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.207+0000 7f9cf331f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:20.222 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.207+0000 7f9cf331f640 1 -- 172.21.15.100:0/2499444836 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cec06eeb0 msgr2=0x7f9cec0a40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:20.222 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.207+0000 7f9cf331f640 1 --2- 172.21.15.100:0/2499444836 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cec06eeb0 0x7f9cec0a40f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc009870 tx=0x7f9cdc02a0a0 comp rx=0 tx=0).stop 2024-09-13T11:53:20.222 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.207+0000 7f9cf331f640 1 -- 172.21.15.100:0/2499444836 shutdown_connections 2024-09-13T11:53:20.222 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.207+0000 7f9cf331f640 1 --2- 172.21.15.100:0/2499444836 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cec06eeb0 0x7f9cec0a40f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:20.222 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.207+0000 7f9cf331f640 1 --2- 172.21.15.100:0/2499444836 >> v2:172.21.15.122:3300/0 conn(0x7f9cec0a37d0 0x7f9cec0a3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:20.222 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.207+0000 7f9cf331f640 1 -- 172.21.15.100:0/2499444836 >> 172.21.15.100:0/2499444836 conn(0x7f9cec0a83f0 msgr2=0x7f9cec0a48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:20.222 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.207+0000 7f9cf331f640 1 -- 172.21.15.100:0/2499444836 shutdown_connections 2024-09-13T11:53:20.222 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.207+0000 7f9cf331f640 1 -- 172.21.15.100:0/2499444836 wait complete. 2024-09-13T11:53:20.222 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.207+0000 7f9cf331f640 10 monclient: build_initial_monmap 2024-09-13T11:53:20.222 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.207+0000 7f9cf331f640 10 monclient: monmap: 2024-09-13T11:53:20.223 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:20.223 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:20.223 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:20.223 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:20.223 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:20.223 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:20.223 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:20.223 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:20.223 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:20.223 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:20.223 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf331f640 1 Processor -- start 2024-09-13T11:53:20.223 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf331f640 1 -- start start 2024-09-13T11:53:20.223 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf331f640 10 monclient: init 2024-09-13T11:53:20.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf331f640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:20.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf331f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:53:20.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf331f640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9cec06eeb0 0x7f9cec1ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:20.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf331f640 10 monclient(hunting): picked mon.noname-a con 0x7f9cec06eeb0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:20.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf331f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cec0a37d0 0x7f9cec1cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:20.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf331f640 10 monclient(hunting): picked mon.noname-b con 0x7f9cec0a37d0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:20.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf331f640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cec137970 0x7f9cec124270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:20.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf331f640 10 monclient(hunting): picked mon.noname-c con 0x7f9cec137970 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:20.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf331f640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:20.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf331f640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec138530 con 0x7f9cec06eeb0 2024-09-13T11:53:20.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf331f640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:20.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf331f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec0a4630 con 0x7f9cec0a37d0 2024-09-13T11:53:20.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf331f640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:20.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf10bb640 1 -- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9cec06eeb0 msgr2=0x7f9cec1ccc70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:3300/0 2024-09-13T11:53:20.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf331f640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f9cec1386b0 con 0x7f9cec137970 2024-09-13T11:53:20.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf331f640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:20.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf331f640 10 monclient(hunting): authenticate will time out at 780.801671s 2024-09-13T11:53:20.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf10bb640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9cec06eeb0 0x7f9cec1ccc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:53:20.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf18bc640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cec137970 0x7f9cec124270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:20.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf18bc640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cec137970 0x7f9cec124270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:35342/0 (socket says 172.21.15.100:35342) 2024-09-13T11:53:20.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf18bc640 1 -- 172.21.15.100:0/2749664165 learned_addr learned my addr 172.21.15.100:0/2749664165 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:20.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf18bc640 10 monclient(hunting): get_auth_request con 0x7f9cec137970 auth_method 0 2024-09-13T11:53:20.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf18bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:20.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf18bc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:20.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf18bc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:20.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf08ba640 1 --2- 172.21.15.100:0/2749664165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cec0a37d0 0x7f9cec1cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:20.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf18bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:20.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf18bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:20.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf18bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:20.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf08ba640 10 monclient(hunting): get_auth_request con 0x7f9cec0a37d0 auth_method 0 2024-09-13T11:53:20.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf08ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:20.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf08ba640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:20.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf08ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:20.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf08ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:20.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf08ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:20.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.208+0000 7f9cf08ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:20.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9cf18bc640 10 monclient(hunting): handle_auth_done global_id 4427 payload 824 2024-09-13T11:53:20.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9cf18bc640 1 -- 172.21.15.100:0/2749664165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cec0a37d0 msgr2=0x7f9cec1cd1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:20.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9cf18bc640 1 --2- 172.21.15.100:0/2749664165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cec0a37d0 0x7f9cec1cd1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:20.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9cf18bc640 1 -- 172.21.15.100:0/2749664165 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9cec06eeb0 msgr2=0x7f9cec1ccc70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:53:20.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9cf18bc640 1 --2- 172.21.15.100:0/2749664165 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9cec06eeb0 0x7f9cec1ccc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:20.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9cf18bc640 10 monclient: _finish_hunting 0 2024-09-13T11:53:20.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9cf18bc640 1 monclient: found mon.noname-c 2024-09-13T11:53:20.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9cf18bc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:20.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9cf18bc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:53:20.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9cf18bc640 1 -- 172.21.15.100:0/2749664165 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cec127db0 con 0x7f9cec137970 2024-09-13T11:53:20.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9cf18bc640 1 --2- 172.21.15.100:0/2749664165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cec137970 0x7f9cec124270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9ce400be60 tx=0x7f9ce400bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:20.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9ce1ffb640 1 -- 172.21.15.100:0/2749664165 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce4016430 con 0x7f9cec137970 2024-09-13T11:53:20.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9ce1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:20.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9ce1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:20.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9ce1ffb640 10 monclient: dump: 2024-09-13T11:53:20.227 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:20.227 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:20.228 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:20.228 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:20.228 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:20.228 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:20.228 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:20.228 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:20.228 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:20.228 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:20.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9ce1ffb640 10 monclient: _finish_auth 0 2024-09-13T11:53:20.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9ce1ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:53:20.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9ce1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:20.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9cf331f640 5 monclient: authenticate success, global_id 4427 2024-09-13T11:53:20.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9ce1ffb640 1 -- 172.21.15.100:0/2749664165 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ce4010030 con 0x7f9cec137970 2024-09-13T11:53:20.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9ce1ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:20.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9ce1ffb640 1 -- 172.21.15.100:0/2749664165 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce4016850 con 0x7f9cec137970 2024-09-13T11:53:20.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9cf331f640 10 monclient: _renew_subs 2024-09-13T11:53:20.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9cf331f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:20.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9cf331f640 1 -- 172.21.15.100:0/2749664165 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cec09f0b0 con 0x7f9cec137970 2024-09-13T11:53:20.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9ce1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:20.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9ce1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:20.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9ce1ffb640 10 monclient: dump: 2024-09-13T11:53:20.229 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:20.229 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:20.229 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:20.229 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:20.229 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:20.229 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:20.230 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:20.230 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:20.230 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:20.230 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:20.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9ce37fe640 4 set_mon_vals no callback set 2024-09-13T11:53:20.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9cf331f640 10 monclient: _renew_subs 2024-09-13T11:53:20.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9cf331f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:20.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.209+0000 7f9cf331f640 1 -- 172.21.15.100:0/2749664165 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cec1277f0 con 0x7f9cec137970 2024-09-13T11:53:20.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.210+0000 7f9ce1ffb640 1 -- 172.21.15.100:0/2749664165 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f9ce40153e0 con 0x7f9cec137970 2024-09-13T11:53:20.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.210+0000 7f9ce1ffb640 1 --2- 172.21.15.100:0/2749664165 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9cd003df50 0x7f9cd0040410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:20.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.210+0000 7f9ce1ffb640 1 -- 172.21.15.100:0/2749664165 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4738+0+0 (secure 0 0 0) 0x7f9ce4052bb0 con 0x7f9cec137970 2024-09-13T11:53:20.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.210+0000 7f9cf331f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:20.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.210+0000 7f9cf331f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:20.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.210+0000 7f9cf331f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:20.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.210+0000 7f9cf331f640 1 -- 172.21.15.100:0/2749664165 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cec137e20 con 0x7f9cec137970 2024-09-13T11:53:20.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.210+0000 7f9cf10bb640 1 --2- 172.21.15.100:0/2749664165 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9cd003df50 0x7f9cd0040410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:20.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.210+0000 7f9cf10bb640 10 monclient: get_auth_request con 0x7f9cd003df50 auth_method 0 2024-09-13T11:53:20.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.211+0000 7f9cf10bb640 1 --2- 172.21.15.100:0/2749664165 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9cd003df50 0x7f9cd0040410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9ce80094f0 tx=0x7f9ce8009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:20.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.215+0000 7f9ce1ffb640 1 -- 172.21.15.100:0/2749664165 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f9ce40060d0 con 0x7f9cec137970 2024-09-13T11:53:20.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.215+0000 7f9ce1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:20.232 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.215+0000 7f9ce1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:20.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.456+0000 7f9cf331f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:53:20.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.456+0000 7f9cf331f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:53:20.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.456+0000 7f9cf331f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:20.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.456+0000 7f9cf331f640 1 -- 172.21.15.100:0/2749664165 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9cec1274c0 con 0x7f9cec137970 2024-09-13T11:53:20.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.456+0000 7f9ce1ffb640 1 -- 172.21.15.100:0/2749664165 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f9ce40132e0 con 0x7f9cec137970 2024-09-13T11:53:20.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.456+0000 7f9ce1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:53:20.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.456+0000 7f9ce1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:20.458 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":34,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:53:20.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.458+0000 7f9cf331f640 1 -- 172.21.15.100:0/2749664165 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9cd003df50 msgr2=0x7f9cd0040410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:20.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.458+0000 7f9cf331f640 1 --2- 172.21.15.100:0/2749664165 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9cd003df50 0x7f9cd0040410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9ce80094f0 tx=0x7f9ce8009190 comp rx=0 tx=0).stop 2024-09-13T11:53:20.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.458+0000 7f9cf331f640 10 monclient: shutdown 2024-09-13T11:53:20.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.458+0000 7f9cf331f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:20.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.458+0000 7f9cf331f640 1 -- 172.21.15.100:0/2749664165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cec137970 msgr2=0x7f9cec124270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:20.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.458+0000 7f9cf331f640 1 --2- 172.21.15.100:0/2749664165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cec137970 0x7f9cec124270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9ce400be60 tx=0x7f9ce400bf50 comp rx=0 tx=0).stop 2024-09-13T11:53:20.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.458+0000 7f9cf331f640 1 -- 172.21.15.100:0/2749664165 shutdown_connections 2024-09-13T11:53:20.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.458+0000 7f9cf331f640 1 --2- 172.21.15.100:0/2749664165 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9cd003df50 0x7f9cd0040410 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:20.461 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.458+0000 7f9cf331f640 1 --2- 172.21.15.100:0/2749664165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9cec137970 0x7f9cec124270 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:20.461 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.458+0000 7f9cf331f640 1 --2- 172.21.15.100:0/2749664165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cec0a37d0 0x7f9cec1cd1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:20.461 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.458+0000 7f9cf331f640 1 --2- 172.21.15.100:0/2749664165 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9cec06eeb0 0x7f9cec1ccc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:20.461 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.458+0000 7f9cf331f640 1 -- 172.21.15.100:0/2749664165 >> 172.21.15.100:0/2749664165 conn(0x7f9cec0a83f0 msgr2=0x7f9cec0a7600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:20.461 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.458+0000 7f9cf331f640 1 -- 172.21.15.100:0/2749664165 shutdown_connections 2024-09-13T11:53:20.461 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:20.458+0000 7f9cf331f640 1 -- 172.21.15.100:0/2749664165 wait complete. 2024-09-13T11:53:23.472 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:53:23.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.578+0000 7f88a5364640 10 monclient: get_monmap_and_config 2024-09-13T11:53:23.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.578+0000 7f88a5364640 10 monclient: build_initial_monmap 2024-09-13T11:53:23.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.578+0000 7f88a5364640 10 monclient: monmap: 2024-09-13T11:53:23.580 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:23.580 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:23.580 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:23.579714+0000 2024-09-13T11:53:23.580 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:23.579714+0000 2024-09-13T11:53:23.580 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:23.581 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:23.581 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:23.581 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:23.581 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:23.581 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:23.581 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:23.581 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:23.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.579+0000 7f88a5364640 1 Processor -- start 2024-09-13T11:53:23.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.579+0000 7f88a5364640 1 -- start start 2024-09-13T11:53:23.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.579+0000 7f88a5364640 10 monclient: init 2024-09-13T11:53:23.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f88a5364640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:23.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f88a5364640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-13T11:53:23.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f88a5364640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f88a0137a60 0x7f88a00a2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:23.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f88a5364640 10 monclient(hunting): picked mon.noname-c con 0x7f88a0137a60 addr v2:172.21.15.122:3300/0 2024-09-13T11:53:23.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f88a5364640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f88a0137e40 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:23.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f88a5364640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f88a0138250 0x7f88a00a5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:23.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f88a5364640 10 monclient(hunting): picked mon.noname-a con 0x7f88a0138250 addr v2:172.21.15.100:3300/0 2024-09-13T11:53:23.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f88a5364640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88a00a5be0 con 0x7f88a0137e40 2024-09-13T11:53:23.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f88a5364640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:23.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f88a5364640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f88a00a4410 con 0x7f88a0138250 2024-09-13T11:53:23.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f88a5364640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:23.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f88a5364640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f88a00a76b0 con 0x7f88a0137a60 2024-09-13T11:53:23.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f88a5364640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:23.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f88a5364640 10 monclient(hunting): authenticate will time out at 784.173158s 2024-09-13T11:53:23.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f889e7fc640 1 -- >> v1:172.21.15.100:6789/0 conn(0x7f88a0137e40 legacy=0x7f88a00a4950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6789/0 2024-09-13T11:53:23.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f889f7fe640 1 -- >> v2:172.21.15.100:3300/0 conn(0x7f88a0138250 msgr2=0x7f88a00a5060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.100:3300/0 2024-09-13T11:53:23.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f889f7fe640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f88a0138250 0x7f88a00a5060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:53:23.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f889effd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f88a0137a60 0x7f88a00a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:23.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f889effd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f88a0137a60 0x7f88a00a2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.100:51788/0 (socket says 172.21.15.100:51788) 2024-09-13T11:53:23.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f889effd640 1 -- 172.21.15.100:0/1981775975 learned_addr learned my addr 172.21.15.100:0/1981775975 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:23.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f889effd640 10 monclient(hunting): get_auth_request con 0x7f88a0137a60 auth_method 0 2024-09-13T11:53:23.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f889effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:23.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f889effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:23.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f889effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:23.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f889effd640 1 -- 172.21.15.100:0/1981775975 >> v2:172.21.15.122:3300/0 conn(0x7f88a0137a60 msgr2=0x7f88a00a2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:53:23.586 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f889effd640 1 -- 172.21.15.100:0/1981775975 >> v2:172.21.15.122:3300/0 conn(0x7f88a0137a60 msgr2=0x7f88a00a2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:53:23.586 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f889effd640 1 --2- 172.21.15.100:0/1981775975 >> v2:172.21.15.122:3300/0 conn(0x7f88a0137a60 0x7f88a00a2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:53:23.586 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.580+0000 7f889effd640 1 --2- 172.21.15.100:0/1981775975 >> v2:172.21.15.122:3300/0 conn(0x7f88a0137a60 0x7f88a00a2980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:53:23.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.781+0000 7f889e7fc640 1 -- 172.21.15.100:0/1981775975 >> v1:172.21.15.100:6789/0 conn(0x7f88a0137e40 legacy=0x7f88a00a4950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6789/0 2024-09-13T11:53:23.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.781+0000 7f889f7fe640 1 -- 172.21.15.100:0/1981775975 >> v2:172.21.15.100:3300/0 conn(0x7f88a0138250 msgr2=0x7f88a00a5060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.100:3300/0 2024-09-13T11:53:23.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.781+0000 7f889f7fe640 1 --2- 172.21.15.100:0/1981775975 >> v2:172.21.15.100:3300/0 conn(0x7f88a0138250 0x7f88a00a5060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T11:53:23.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.782+0000 7f889effd640 1 --2- 172.21.15.100:0/1981775975 >> v2:172.21.15.122:3300/0 conn(0x7f88a0137a60 0x7f88a00a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:23.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.782+0000 7f889effd640 10 monclient(hunting): get_auth_request con 0x7f88a0137a60 auth_method 0 2024-09-13T11:53:23.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.782+0000 7f889effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:23.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.782+0000 7f889effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:23.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.782+0000 7f889effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:53:23.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.782+0000 7f889effd640 1 -- 172.21.15.100:0/1981775975 >> v2:172.21.15.122:3300/0 conn(0x7f88a0137a60 msgr2=0x7f88a00a2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:53:23.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.782+0000 7f889effd640 1 -- 172.21.15.100:0/1981775975 >> v2:172.21.15.122:3300/0 conn(0x7f88a0137a60 msgr2=0x7f88a00a2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:53:23.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.782+0000 7f889effd640 1 --2- 172.21.15.100:0/1981775975 >> v2:172.21.15.122:3300/0 conn(0x7f88a0137a60 0x7f88a00a2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:53:23.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:23.782+0000 7f889effd640 1 --2- 172.21.15.100:0/1981775975 >> v2:172.21.15.122:3300/0 conn(0x7f88a0137a60 0x7f88a00a2980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T11:53:24.183 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:24.182+0000 7f889e7fc640 1 -- 172.21.15.100:0/1981775975 >> v1:172.21.15.100:6789/0 conn(0x7f88a0137e40 legacy=0x7f88a00a4950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6789/0 2024-09-13T11:53:24.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:24.182+0000 7f889f7fe640 1 -- 172.21.15.100:0/1981775975 >> v2:172.21.15.100:3300/0 conn(0x7f88a0138250 msgr2=0x7f88a00a5060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.100:3300/0 2024-09-13T11:53:24.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:24.182+0000 7f889f7fe640 1 --2- 172.21.15.100:0/1981775975 >> v2:172.21.15.100:3300/0 conn(0x7f88a0138250 0x7f88a00a5060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T11:53:24.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:24.184+0000 7f889effd640 1 --2- 172.21.15.100:0/1981775975 >> v2:172.21.15.122:3300/0 conn(0x7f88a0137a60 0x7f88a00a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:24.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:24.184+0000 7f889effd640 10 monclient(hunting): get_auth_request con 0x7f88a0137a60 auth_method 0 2024-09-13T11:53:24.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:24.184+0000 7f889effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:24.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:24.184+0000 7f889effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:24.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:24.184+0000 7f889effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:53:24.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:24.184+0000 7f889effd640 1 -- 172.21.15.100:0/1981775975 >> v2:172.21.15.122:3300/0 conn(0x7f88a0137a60 msgr2=0x7f88a00a2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:53:24.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:24.184+0000 7f889effd640 1 -- 172.21.15.100:0/1981775975 >> v2:172.21.15.122:3300/0 conn(0x7f88a0137a60 msgr2=0x7f88a00a2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:53:24.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:24.184+0000 7f889effd640 1 --2- 172.21.15.100:0/1981775975 >> v2:172.21.15.122:3300/0 conn(0x7f88a0137a60 0x7f88a00a2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:53:24.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:24.184+0000 7f889effd640 1 --2- 172.21.15.100:0/1981775975 >> v2:172.21.15.122:3300/0 conn(0x7f88a0137a60 0x7f88a00a2980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T11:53:24.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:24.983+0000 7f889f7fe640 1 -- 172.21.15.100:0/1981775975 >> v2:172.21.15.100:3300/0 conn(0x7f88a0138250 msgr2=0x7f88a00a5060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.100:3300/0 2024-09-13T11:53:24.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:24.983+0000 7f889e7fc640 1 -- 172.21.15.100:0/1981775975 >> v1:172.21.15.100:6789/0 conn(0x7f88a0137e40 legacy=0x7f88a00a4950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6789/0 2024-09-13T11:53:24.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:24.983+0000 7f889f7fe640 1 --2- 172.21.15.100:0/1981775975 >> v2:172.21.15.100:3300/0 conn(0x7f88a0138250 0x7f88a00a5060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T11:53:24.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:24.985+0000 7f889effd640 1 --2- 172.21.15.100:0/1981775975 >> v2:172.21.15.122:3300/0 conn(0x7f88a0137a60 0x7f88a00a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:24.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:24.985+0000 7f889effd640 10 monclient(hunting): get_auth_request con 0x7f88a0137a60 auth_method 0 2024-09-13T11:53:24.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:24.985+0000 7f889effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:24.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:24.985+0000 7f889effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:24.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:24.985+0000 7f889effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:53:24.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:24.985+0000 7f889effd640 1 -- 172.21.15.100:0/1981775975 >> v2:172.21.15.122:3300/0 conn(0x7f88a0137a60 msgr2=0x7f88a00a2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:53:24.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:24.985+0000 7f889effd640 1 -- 172.21.15.100:0/1981775975 >> v2:172.21.15.122:3300/0 conn(0x7f88a0137a60 msgr2=0x7f88a00a2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:53:24.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:24.985+0000 7f889effd640 1 --2- 172.21.15.100:0/1981775975 >> v2:172.21.15.122:3300/0 conn(0x7f88a0137a60 0x7f88a00a2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:53:24.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:24.985+0000 7f889effd640 1 --2- 172.21.15.100:0/1981775975 >> v2:172.21.15.122:3300/0 conn(0x7f88a0137a60 0x7f88a00a2980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T11:53:26.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889cff9640 10 monclient(hunting): tick 2024-09-13T11:53:26.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889cff9640 10 monclient(hunting): _check_auth_tickets 2024-09-13T11:53:26.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889cff9640 1 monclient(hunting): continuing hunt 2024-09-13T11:53:26.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889cff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-13T11:53:26.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889cff9640 1 -- 172.21.15.100:0/1981775975 >> v2:172.21.15.122:3300/0 conn(0x7f88a0137a60 msgr2=0x7f88a00a2980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:53:26.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889cff9640 1 --2- 172.21.15.100:0/1981775975 >> v2:172.21.15.122:3300/0 conn(0x7f88a0137a60 0x7f88a00a2980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:26.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889cff9640 1 -- 172.21.15.100:0/1981775975 >> v2:172.21.15.100:3300/0 conn(0x7f88a0138250 msgr2=0x7f88a00a5060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:53:26.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889cff9640 1 --2- 172.21.15.100:0/1981775975 >> v2:172.21.15.100:3300/0 conn(0x7f88a0138250 0x7f88a00a5060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:26.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889cff9640 1 -- 172.21.15.100:0/1981775975 >> v1:172.21.15.100:6789/0 conn(0x7f88a0137e40 legacy=0x7f88a00a4950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:53:26.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889cff9640 10 monclient: _add_conns ranks=[1,2] 2024-09-13T11:53:26.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889cff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8880003690 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:26.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889cff9640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8880007360 0x7f8880009820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:26.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889cff9640 10 monclient(hunting): picked mon.noname-b con 0x7f8880007360 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:26.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889cff9640 1 -- 172.21.15.100:0/1981775975 --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88a00a19e0 con 0x7f8880003690 2024-09-13T11:53:26.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889cff9640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:26.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889cff9640 1 -- 172.21.15.100:0/1981775975 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f88a00a76b0 con 0x7f8880007360 2024-09-13T11:53:26.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889cff9640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:26.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889e7fc640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8880007360 0x7f8880009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:26.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889e7fc640 10 monclient(hunting): get_auth_request con 0x7f8880007360 auth_method 0 2024-09-13T11:53:26.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:26.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:26.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:26.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889e7fc640 1 -- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8880007360 msgr2=0x7f8880009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T11:53:26.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889e7fc640 1 -- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8880007360 msgr2=0x7f8880009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:53:26.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889e7fc640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8880007360 0x7f8880009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:53:26.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.580+0000 7f889e7fc640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8880007360 0x7f8880009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:53:26.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.782+0000 7f889e7fc640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8880007360 0x7f8880009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:26.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.782+0000 7f889e7fc640 10 monclient(hunting): get_auth_request con 0x7f8880007360 auth_method 0 2024-09-13T11:53:26.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.782+0000 7f889e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:26.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.782+0000 7f889e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:26.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.782+0000 7f889e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:53:26.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.782+0000 7f889e7fc640 1 -- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8880007360 msgr2=0x7f8880009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T11:53:26.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.782+0000 7f889e7fc640 1 -- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8880007360 msgr2=0x7f8880009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:53:26.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.782+0000 7f889e7fc640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8880007360 0x7f8880009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:53:26.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:26.782+0000 7f889e7fc640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8880007360 0x7f8880009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T11:53:27.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:27.183+0000 7f889e7fc640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8880007360 0x7f8880009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:27.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:27.183+0000 7f889e7fc640 10 monclient(hunting): get_auth_request con 0x7f8880007360 auth_method 0 2024-09-13T11:53:27.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:27.183+0000 7f889e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:27.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:27.183+0000 7f889e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:27.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:27.183+0000 7f889e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:53:27.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:27.183+0000 7f889e7fc640 1 -- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8880007360 msgr2=0x7f8880009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T11:53:27.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:27.183+0000 7f889e7fc640 1 -- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8880007360 msgr2=0x7f8880009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:53:27.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:27.183+0000 7f889e7fc640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8880007360 0x7f8880009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:53:27.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:27.183+0000 7f889e7fc640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8880007360 0x7f8880009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T11:53:27.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:27.984+0000 7f889e7fc640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8880007360 0x7f8880009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:27.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:27.984+0000 7f889e7fc640 10 monclient(hunting): get_auth_request con 0x7f8880007360 auth_method 0 2024-09-13T11:53:27.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:27.984+0000 7f889e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:27.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:27.984+0000 7f889e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:27.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:27.984+0000 7f889e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:53:27.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:27.984+0000 7f889e7fc640 1 -- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8880007360 msgr2=0x7f8880009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T11:53:27.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:27.984+0000 7f889e7fc640 1 -- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8880007360 msgr2=0x7f8880009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:53:27.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:27.984+0000 7f889e7fc640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8880007360 0x7f8880009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:53:27.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:27.984+0000 7f889e7fc640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8880007360 0x7f8880009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T11:53:28.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.294+0000 7f889dffb640 1 -- 172.21.15.100:0/1981775975 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3620005458 0 0) 0x7f88a00a19e0 con 0x7f8880003690 2024-09-13T11:53:28.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.294+0000 7f889dffb640 10 monclient(hunting): _negotiate 2024-09-13T11:53:28.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.294+0000 7f889dffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:28.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.294+0000 7f889dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:28.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.294+0000 7f889dffb640 10 monclient(hunting): my global_id is 4417 2024-09-13T11:53:28.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.294+0000 7f889dffb640 1 -- 172.21.15.100:0/1981775975 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f888c003520 con 0x7f8880003690 2024-09-13T11:53:28.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 1 -- 172.21.15.100:0/1981775975 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3886442955 0 0) 0x7f888c003520 con 0x7f8880003690 2024-09-13T11:53:28.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 1 -- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8880007360 msgr2=0x7f8880009820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:53:28.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8880007360 0x7f8880009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 10 monclient: _finish_hunting 0 2024-09-13T11:53:28.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 1 monclient: found mon.noname-c-legacy 2024-09-13T11:53:28.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:28.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.122:6789/0 2024-09-13T11:53:28.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 1 -- 172.21.15.100:0/1981775975 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a00a5be0 con 0x7f8880003690 2024-09-13T11:53:28.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 10 monclient: _finish_auth 0 2024-09-13T11:53:28.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 10 monclient: _check_auth_tickets 2024-09-13T11:53:28.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:28.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 1 -- 172.21.15.100:0/1981775975 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 626700363 0 0) 0x7f88900073f0 con 0x7f8880003690 2024-09-13T11:53:28.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:28.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-13T11:53:28.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 10 monclient: dump: 2024-09-13T11:53:28.300 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:28.301 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:28.301 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.301 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.301 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:28.301 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:28.301 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:28.301 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:28.301 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:28.301 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:28.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] but i'm connected to v1:172.21.15.122:6789/0, reconnecting 2024-09-13T11:53:28.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:28.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 1 -- 172.21.15.100:0/1981775975 >> v1:172.21.15.122:6789/0 conn(0x7f8880003690 legacy=0x7f8880005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:28.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:53:28.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889f7fe640 1 -- 172.21.15.100:0/1981775975 reap_dead start 2024-09-13T11:53:28.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f888c0039b0 0x7f888c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:28.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 10 monclient(hunting): picked mon.a con 0x7f888c0039b0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:28.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f888c004800 0x7f888c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:28.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 10 monclient(hunting): picked mon.c con 0x7f888c004800 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:28.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f888c0052e0 0x7f888c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:28.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 10 monclient(hunting): picked mon.b con 0x7f888c0052e0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:28.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:28.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 1 -- 172.21.15.100:0/1981775975 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f88a00a76b0 con 0x7f888c0039b0 2024-09-13T11:53:28.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:28.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 1 -- 172.21.15.100:0/1981775975 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f888c00c190 con 0x7f888c0052e0 2024-09-13T11:53:28.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:28.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 1 -- 172.21.15.100:0/1981775975 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f888c00c390 con 0x7f888c004800 2024-09-13T11:53:28.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:28.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889dffb640 10 monclient(hunting): _finish_auth 0 2024-09-13T11:53:28.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889effd640 1 -- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f888c0039b0 msgr2=0x7f888c003db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.100:3300/0 2024-09-13T11:53:28.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889effd640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f888c0039b0 0x7f888c003db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:53:28.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889e7fc640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f888c004800 0x7f888c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:28.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889e7fc640 10 monclient(hunting): get_auth_request con 0x7f888c004800 auth_method 0 2024-09-13T11:53:28.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:28.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:28.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:53:28.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889f7fe640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f888c0052e0 0x7f888c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:28.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:28.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:28.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:53:28.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889f7fe640 10 monclient(hunting): get_auth_request con 0x7f888c0052e0 auth_method 0 2024-09-13T11:53:28.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:28.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:28.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:53:28.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:28.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:28.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:53:28.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889e7fc640 10 monclient(hunting): handle_auth_done global_id 4417 payload 293 2024-09-13T11:53:28.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889e7fc640 1 -- 172.21.15.100:0/1981775975 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f888c0052e0 msgr2=0x7f888c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:28.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889e7fc640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f888c0052e0 0x7f888c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889e7fc640 1 -- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f888c0039b0 msgr2=0x7f888c003db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:53:28.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889e7fc640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f888c0039b0 0x7f888c003db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889e7fc640 10 monclient: _finish_hunting 0 2024-09-13T11:53:28.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889e7fc640 1 monclient: found mon.c 2024-09-13T11:53:28.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:28.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889e7fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:28.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889e7fc640 1 -- 172.21.15.100:0/1981775975 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f888c003520 con 0x7f888c004800 2024-09-13T11:53:28.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889e7fc640 10 monclient: _finish_auth 0 2024-09-13T11:53:28.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889e7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:53:28.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f889e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:28.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f88a5364640 5 monclient: authenticate success, global_id 4417 2024-09-13T11:53:28.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.296+0000 7f88a5364640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:28.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.297+0000 7f889e7fc640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f888c004800 0x7f888c004c80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8888003980 tx=0x7f888c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85072b231f27b51d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:28.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.299+0000 7f889dffb640 1 -- 172.21.15.100:0/1981775975 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8888039070 con 0x7f888c004800 2024-09-13T11:53:28.311 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.299+0000 7f889dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:28.311 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.299+0000 7f889dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:28.311 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.299+0000 7f889dffb640 10 monclient: dump: 2024-09-13T11:53:28.311 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:28.311 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:28.311 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.311 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.311 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:28.311 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:28.311 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:28.311 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:28.311 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:28.312 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:28.312 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.299+0000 7f88a5364640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:28.312 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.299+0000 7f889dffb640 1 -- 172.21.15.100:0/1981775975 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8888034410 con 0x7f888c004800 2024-09-13T11:53:28.312 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.299+0000 7f889dffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:28.312 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.299+0000 7f88a5364640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:28.312 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.299+0000 7f88a5364640 4 set_mon_vals no callback set 2024-09-13T11:53:28.312 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.299+0000 7f889dffb640 1 -- 172.21.15.100:0/1981775975 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8888009ad0 con 0x7f888c004800 2024-09-13T11:53:28.312 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.299+0000 7f889dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:28.312 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.299+0000 7f889dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:28.312 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.299+0000 7f889dffb640 10 monclient: dump: 2024-09-13T11:53:28.312 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:28.312 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:28.312 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.313 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.313 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:28.313 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:28.313 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:28.313 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:28.313 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:28.313 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:28.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.300+0000 7f88a5364640 10 monclient: shutdown 2024-09-13T11:53:28.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.300+0000 7f88a5364640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:28.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.300+0000 7f88a5364640 1 -- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f888c004800 msgr2=0x7f888c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:28.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.300+0000 7f88a5364640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f888c004800 0x7f888c004c80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8888003980 tx=0x7f888c005830 comp rx=0 tx=0).stop 2024-09-13T11:53:28.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.300+0000 7f88a5364640 1 -- 172.21.15.100:0/1981775975 shutdown_connections 2024-09-13T11:53:28.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.300+0000 7f88a5364640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f888c0052e0 0x7f888c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.300+0000 7f88a5364640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f888c004800 0x7f888c004c80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.300+0000 7f88a5364640 1 --2- 172.21.15.100:0/1981775975 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f888c0039b0 0x7f888c003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.300+0000 7f88a5364640 1 -- 172.21.15.100:0/1981775975 >> 172.21.15.100:0/1981775975 conn(0x7f88a00aa580 msgr2=0x7f88a00a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:28.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.300+0000 7f88a5364640 1 -- 172.21.15.100:0/1981775975 shutdown_connections 2024-09-13T11:53:28.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.300+0000 7f88a5364640 1 -- 172.21.15.100:0/1981775975 wait complete. 2024-09-13T11:53:28.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.301+0000 7f88a5364640 10 monclient: build_initial_monmap 2024-09-13T11:53:28.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.301+0000 7f88a5364640 10 monclient: monmap: 2024-09-13T11:53:28.314 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:28.314 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:28.314 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:28.314 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:28.314 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:28.314 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:28.315 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:28.315 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:28.315 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:28.315 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:28.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f88a5364640 1 Processor -- start 2024-09-13T11:53:28.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f88a5364640 1 -- start start 2024-09-13T11:53:28.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f88a5364640 10 monclient: init 2024-09-13T11:53:28.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f88a5364640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:28.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f88a5364640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:53:28.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f88a5364640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f888c0039b0 0x7f88a01cc770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:28.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f88a5364640 10 monclient(hunting): picked mon.noname-c con 0x7f888c0039b0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:28.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f88a5364640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f888c004800 0x7f88a01cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:28.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f88a5364640 10 monclient(hunting): picked mon.noname-a con 0x7f888c004800 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:28.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f88a5364640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f888c0052e0 0x7f88a01240a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:28.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f88a5364640 10 monclient(hunting): picked mon.noname-b con 0x7f888c0052e0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:28.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f88a5364640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:28.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f88a5364640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f88a006ff00 con 0x7f888c004800 2024-09-13T11:53:28.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f88a5364640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:28.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f88a5364640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f88a006fd80 con 0x7f888c0052e0 2024-09-13T11:53:28.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f88a5364640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:28.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f88a5364640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f88a00a2980 con 0x7f888c0039b0 2024-09-13T11:53:28.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f88a5364640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:28.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f88a5364640 10 monclient(hunting): authenticate will time out at 788.895759s 2024-09-13T11:53:28.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f889e7fc640 1 -- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f888c004800 msgr2=0x7f88a01cccb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:3300/0 2024-09-13T11:53:28.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f889e7fc640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f888c004800 0x7f88a01cccb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:53:28.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f889effd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f888c0039b0 0x7f88a01cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:28.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f889effd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f888c0039b0 0x7f88a01cc770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:36886/0 (socket says 172.21.15.100:36886) 2024-09-13T11:53:28.318 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f889effd640 1 -- 172.21.15.100:0/1185835949 learned_addr learned my addr 172.21.15.100:0/1185835949 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:28.318 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f889effd640 10 monclient(hunting): get_auth_request con 0x7f888c0039b0 auth_method 0 2024-09-13T11:53:28.318 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f889effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:28.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f889effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:28.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f889effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:28.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f889effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:28.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f889effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:28.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.302+0000 7f889effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:28.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f889f7fe640 1 --2- 172.21.15.100:0/1185835949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f888c0052e0 0x7f88a01240a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:28.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f889f7fe640 10 monclient(hunting): get_auth_request con 0x7f888c0052e0 auth_method 0 2024-09-13T11:53:28.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f889f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:28.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f889f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:28.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f889f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:28.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f889f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:28.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f889f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:28.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f889f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:28.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f889effd640 10 monclient(hunting): handle_auth_done global_id 4430 payload 824 2024-09-13T11:53:28.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f889effd640 1 -- 172.21.15.100:0/1185835949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f888c0052e0 msgr2=0x7f88a01240a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:28.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f889effd640 1 --2- 172.21.15.100:0/1185835949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f888c0052e0 0x7f88a01240a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f889effd640 1 -- 172.21.15.100:0/1185835949 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f888c004800 msgr2=0x7f88a01cccb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:53:28.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f889effd640 1 --2- 172.21.15.100:0/1185835949 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f888c004800 0x7f88a01cccb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f889effd640 10 monclient: _finish_hunting 0 2024-09-13T11:53:28.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f889effd640 1 monclient: found mon.noname-c 2024-09-13T11:53:28.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f889effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:28.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f889effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:53:28.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f889effd640 1 -- 172.21.15.100:0/1185835949 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a00a6370 con 0x7f888c0039b0 2024-09-13T11:53:28.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f889effd640 1 --2- 172.21.15.100:0/1185835949 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f888c0039b0 0x7f88a01cc770 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f889000bfd0 tx=0x7f88900075c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:28.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f88977fe640 1 -- 172.21.15.100:0/1185835949 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889001a070 con 0x7f888c0039b0 2024-09-13T11:53:28.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f88977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:28.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f88977fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:28.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f88977fe640 10 monclient: dump: 2024-09-13T11:53:28.321 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:28.321 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:28.321 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.321 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.321 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:28.321 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:28.321 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:28.321 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:28.321 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:28.321 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:28.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f88977fe640 10 monclient: _finish_auth 0 2024-09-13T11:53:28.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f88977fe640 10 monclient: _check_auth_tickets 2024-09-13T11:53:28.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f88977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:28.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f88977fe640 1 -- 172.21.15.100:0/1185835949 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f889000b2f0 con 0x7f888c0039b0 2024-09-13T11:53:28.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f88a5364640 5 monclient: authenticate success, global_id 4430 2024-09-13T11:53:28.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f88a5364640 10 monclient: _renew_subs 2024-09-13T11:53:28.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f88a5364640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:28.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f88a5364640 1 -- 172.21.15.100:0/1185835949 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88a0125b20 con 0x7f888c0039b0 2024-09-13T11:53:28.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f88977fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:28.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f88977fe640 1 -- 172.21.15.100:0/1185835949 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8890003990 con 0x7f888c0039b0 2024-09-13T11:53:28.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f88977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:28.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f88977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:28.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f88977fe640 10 monclient: dump: 2024-09-13T11:53:28.322 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:28.322 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:28.323 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.323 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.323 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:28.323 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:28.323 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:28.323 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:28.323 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:28.323 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:28.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f889d7fa640 4 set_mon_vals no callback set 2024-09-13T11:53:28.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f88a5364640 10 monclient: _renew_subs 2024-09-13T11:53:28.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f88a5364640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:28.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.303+0000 7f88a5364640 1 -- 172.21.15.100:0/1185835949 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f88a0126680 con 0x7f888c0039b0 2024-09-13T11:53:28.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.304+0000 7f88977fe640 1 -- 172.21.15.100:0/1185835949 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f8890005d10 con 0x7f888c0039b0 2024-09-13T11:53:28.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.304+0000 7f88977fe640 1 --2- 172.21.15.100:0/1185835949 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f888003e680 0x7f8880040b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:28.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.304+0000 7f88977fe640 1 -- 172.21.15.100:0/1185835949 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4738+0+0 (secure 0 0 0) 0x7f8890053560 con 0x7f888c0039b0 2024-09-13T11:53:28.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.305+0000 7f88a5364640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:28.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.305+0000 7f88a5364640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:28.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.305+0000 7f88a5364640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:28.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.305+0000 7f88a5364640 1 -- 172.21.15.100:0/1185835949 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88a00a19e0 con 0x7f888c0039b0 2024-09-13T11:53:28.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.305+0000 7f889e7fc640 1 --2- 172.21.15.100:0/1185835949 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f888003e680 0x7f8880040b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:28.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.305+0000 7f889e7fc640 10 monclient: get_auth_request con 0x7f888003e680 auth_method 0 2024-09-13T11:53:28.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.305+0000 7f889e7fc640 1 --2- 172.21.15.100:0/1185835949 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f888003e680 0x7f8880040b40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8888036000 tx=0x7f888802f000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:28.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.309+0000 7f88977fe640 1 -- 172.21.15.100:0/1185835949 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f8890023020 con 0x7f888c0039b0 2024-09-13T11:53:28.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.310+0000 7f88977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:28.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.310+0000 7f88977fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:28.451 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34553]: starting. 2024-09-13T11:53:28.451 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34553]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-34552 2024-09-13T11:53:28.451 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34553]: created object 0... 2024-09-13T11:53:28.451 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34553]: created object 25... 2024-09-13T11:53:28.452 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34553]: created object 49... 2024-09-13T11:53:28.452 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34553]: finishing. 2024-09-13T11:53:28.452 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34553]: shutting down. 2024-09-13T11:53:28.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.575+0000 7f88a5364640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:53:28.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.575+0000 7f88a5364640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:53:28.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.575+0000 7f88a5364640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:28.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.575+0000 7f88a5364640 1 -- 172.21.15.100:0/1185835949 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f88a00a10c0 con 0x7f888c0039b0 2024-09-13T11:53:28.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.576+0000 7f88977fe640 1 -- 172.21.15.100:0/1185835949 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f8890014d30 con 0x7f888c0039b0 2024-09-13T11:53:28.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.576+0000 7f88977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:53:28.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.576+0000 7f88977fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:28.578 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":12,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:53:28.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.578+0000 7f88a5364640 1 -- 172.21.15.100:0/1185835949 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f888003e680 msgr2=0x7f8880040b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:28.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.578+0000 7f88a5364640 1 --2- 172.21.15.100:0/1185835949 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f888003e680 0x7f8880040b40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8888036000 tx=0x7f888802f000 comp rx=0 tx=0).stop 2024-09-13T11:53:28.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.578+0000 7f88a5364640 10 monclient: shutdown 2024-09-13T11:53:28.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.578+0000 7f88a5364640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:28.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.578+0000 7f88a5364640 1 -- 172.21.15.100:0/1185835949 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f888c0039b0 msgr2=0x7f88a01cc770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:28.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.578+0000 7f88a5364640 1 --2- 172.21.15.100:0/1185835949 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f888c0039b0 0x7f88a01cc770 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f889000bfd0 tx=0x7f88900075c0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.578+0000 7f88a5364640 1 -- 172.21.15.100:0/1185835949 shutdown_connections 2024-09-13T11:53:28.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.578+0000 7f88a5364640 1 --2- 172.21.15.100:0/1185835949 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f888003e680 0x7f8880040b40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.578+0000 7f88a5364640 1 --2- 172.21.15.100:0/1185835949 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f888c004800 0x7f88a01cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.578+0000 7f88a5364640 1 --2- 172.21.15.100:0/1185835949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f888c0052e0 0x7f88a01240a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.578+0000 7f88a5364640 1 --2- 172.21.15.100:0/1185835949 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f888c0039b0 0x7f88a01cc770 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.578+0000 7f88a5364640 1 -- 172.21.15.100:0/1185835949 >> 172.21.15.100:0/1185835949 conn(0x7f88a00aa580 msgr2=0x7f88a00a2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:28.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.578+0000 7f88a5364640 1 -- 172.21.15.100:0/1185835949 shutdown_connections 2024-09-13T11:53:28.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.578+0000 7f88a5364640 1 -- 172.21.15.100:0/1185835949 wait complete. 2024-09-13T11:53:28.591 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-13T11:53:28.591 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-13T11:53:28.592 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T11:53:28.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.688+0000 7fc5c7aef640 10 monclient: get_monmap_and_config 2024-09-13T11:53:28.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.688+0000 7fc5c7aef640 10 monclient: build_initial_monmap 2024-09-13T11:53:28.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.688+0000 7fc5c7aef640 10 monclient: monmap: 2024-09-13T11:53:28.690 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:28.690 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:28.690 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:28.689458+0000 2024-09-13T11:53:28.691 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:28.689458+0000 2024-09-13T11:53:28.691 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:28.691 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:28.691 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:28.691 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:28.691 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:28.691 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:28.692 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:28.692 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:28.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.689+0000 7fc5c7aef640 1 Processor -- start 2024-09-13T11:53:28.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.689+0000 7fc5c7aef640 1 -- start start 2024-09-13T11:53:28.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.689+0000 7fc5c7aef640 10 monclient: init 2024-09-13T11:53:28.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.689+0000 7fc5c7aef640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:28.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.689+0000 7fc5c7aef640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-13T11:53:28.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.689+0000 7fc5c7aef640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c00a37d0 0x7fc5c00a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:28.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.689+0000 7fc5c7aef640 10 monclient(hunting): picked mon.noname-b con 0x7fc5c00a37d0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:28.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.689+0000 7fc5c7aef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc5c006eeb0 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:28.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.689+0000 7fc5c7aef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc5c00a40f0 addr v1:172.21.15.100:6789/0 2024-09-13T11:53:28.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.689+0000 7fc5c7aef640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5c0138790 con 0x7fc5c00a40f0 2024-09-13T11:53:28.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.689+0000 7fc5c7aef640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5c00a45a0 con 0x7fc5c006eeb0 2024-09-13T11:53:28.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.689+0000 7fc5c7aef640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:28.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.689+0000 7fc5c7aef640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fc5c009fde0 con 0x7fc5c00a37d0 2024-09-13T11:53:28.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.689+0000 7fc5c7aef640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:28.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.689+0000 7fc5c7aef640 10 monclient(hunting): authenticate will time out at 789.282968s 2024-09-13T11:53:28.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.689+0000 7fc5c608c640 1 -- >> v1:172.21.15.100:6789/0 conn(0x7fc5c00a40f0 legacy=0x7fc5c0138080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6789/0 2024-09-13T11:53:28.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.690+0000 7fc5c588b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c00a37d0 0x7fc5c00a3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:28.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.690+0000 7fc5c588b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c00a37d0 0x7fc5c00a3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:36890/0 (socket says 172.21.15.100:36890) 2024-09-13T11:53:28.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.690+0000 7fc5c588b640 1 -- 172.21.15.100:0/2034976041 learned_addr learned my addr 172.21.15.100:0/2034976041 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:28.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.690+0000 7fc5c588b640 10 monclient(hunting): get_auth_request con 0x7fc5c00a37d0 auth_method 0 2024-09-13T11:53:28.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.690+0000 7fc5c588b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:28.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.690+0000 7fc5c588b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:28.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.690+0000 7fc5c588b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:28.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.690+0000 7fc5c588b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:28.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.690+0000 7fc5c588b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:28.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.690+0000 7fc5c588b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:28.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.690+0000 7fc5c588b640 10 monclient(hunting): handle_auth_done global_id 4433 payload 274 2024-09-13T11:53:28.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.690+0000 7fc5c588b640 1 -- 172.21.15.100:0/2034976041 >> v1:172.21.15.122:6789/0 conn(0x7fc5c006eeb0 legacy=0x7fc5c0137970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:28.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.690+0000 7fc5c588b640 1 -- 172.21.15.100:0/2034976041 >> v1:172.21.15.100:6789/0 conn(0x7fc5c00a40f0 legacy=0x7fc5c0138080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:53:28.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.690+0000 7fc5c588b640 10 monclient: _finish_hunting 0 2024-09-13T11:53:28.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.690+0000 7fc5c588b640 1 monclient: found mon.noname-b 2024-09-13T11:53:28.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.690+0000 7fc5c588b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:28.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.690+0000 7fc5c588b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:53:28.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.690+0000 7fc5c588b640 1 -- 172.21.15.100:0/2034976041 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c006e950 con 0x7fc5c00a37d0 2024-09-13T11:53:28.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c588b640 1 --2- 172.21.15.100:0/2034976041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c00a37d0 0x7fc5c00a3bb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc0098a0 tx=0x7fc5bc02a070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a28ca87476df031a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:28.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c4889640 1 -- 172.21.15.100:0/2034976041 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc037070 con 0x7fc5c00a37d0 2024-09-13T11:53:28.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c4889640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:28.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c4889640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:28.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c4889640 10 monclient: dump: 2024-09-13T11:53:28.704 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:28.704 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:28.705 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.705 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.705 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:28.705 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:28.705 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:28.705 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:28.705 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:28.705 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:28.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c4889640 10 monclient: _finish_auth 0 2024-09-13T11:53:28.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c4889640 10 monclient: _check_auth_tickets 2024-09-13T11:53:28.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c4889640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:28.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c7aef640 5 monclient: authenticate success, global_id 4433 2024-09-13T11:53:28.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c4889640 1 -- 172.21.15.100:0/2034976041 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5bc02adc0 con 0x7fc5c00a37d0 2024-09-13T11:53:28.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c7aef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:28.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c4889640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:28.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c7aef640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:28.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c7aef640 4 set_mon_vals no callback set 2024-09-13T11:53:28.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c4889640 1 -- 172.21.15.100:0/2034976041 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc032660 con 0x7fc5c00a37d0 2024-09-13T11:53:28.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c4889640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:28.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c4889640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:28.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c4889640 10 monclient: dump: 2024-09-13T11:53:28.706 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:28.706 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:28.706 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.706 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.706 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:28.706 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:28.707 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:28.707 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:28.707 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:28.707 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:28.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c7aef640 10 monclient: shutdown 2024-09-13T11:53:28.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c7aef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:28.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c7aef640 1 -- 172.21.15.100:0/2034976041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c00a37d0 msgr2=0x7fc5c00a3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:28.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c7aef640 1 --2- 172.21.15.100:0/2034976041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c00a37d0 0x7fc5c00a3bb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc0098a0 tx=0x7fc5bc02a070 comp rx=0 tx=0).stop 2024-09-13T11:53:28.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c7aef640 1 -- 172.21.15.100:0/2034976041 shutdown_connections 2024-09-13T11:53:28.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c7aef640 1 --2- 172.21.15.100:0/2034976041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c00a37d0 0x7fc5c00a3bb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c7aef640 1 -- 172.21.15.100:0/2034976041 >> 172.21.15.100:0/2034976041 conn(0x7fc5c00a83f0 msgr2=0x7fc5c00a48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:28.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c7aef640 1 -- 172.21.15.100:0/2034976041 shutdown_connections 2024-09-13T11:53:28.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.691+0000 7fc5c7aef640 1 -- 172.21.15.100:0/2034976041 wait complete. 2024-09-13T11:53:28.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.692+0000 7fc5c7aef640 10 monclient: build_initial_monmap 2024-09-13T11:53:28.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.692+0000 7fc5c7aef640 10 monclient: monmap: 2024-09-13T11:53:28.709 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:28.709 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:28.709 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:28.709 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:28.709 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:28.709 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:28.709 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:28.709 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:28.709 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:28.709 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:28.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.692+0000 7fc5c7aef640 1 Processor -- start 2024-09-13T11:53:28.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.692+0000 7fc5c7aef640 1 -- start start 2024-09-13T11:53:28.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.692+0000 7fc5c7aef640 10 monclient: init 2024-09-13T11:53:28.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c7aef640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:28.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c7aef640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:53:28.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c7aef640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c006eeb0 0x7fc5c0137ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:28.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c7aef640 10 monclient(hunting): picked mon.noname-c con 0x7fc5c006eeb0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:28.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c7aef640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5c00a37d0 0x7fc5c01380e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:28.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c7aef640 10 monclient(hunting): picked mon.noname-b con 0x7fc5c00a37d0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:28.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c7aef640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5c00a40f0 0x7fc5c013b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:28.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c7aef640 10 monclient(hunting): picked mon.noname-a con 0x7fc5c00a40f0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:28.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c7aef640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:28.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c7aef640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c009ec50 con 0x7fc5c00a40f0 2024-09-13T11:53:28.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c7aef640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:28.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c7aef640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c009edd0 con 0x7fc5c00a37d0 2024-09-13T11:53:28.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c7aef640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:28.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c7aef640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fc5c0140c50 con 0x7fc5c006eeb0 2024-09-13T11:53:28.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c7aef640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:28.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c7aef640 10 monclient(hunting): authenticate will time out at 789.286234s 2024-09-13T11:53:28.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c608c640 1 -- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5c00a40f0 msgr2=0x7fc5c013b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:3300/0 2024-09-13T11:53:28.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c588b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c006eeb0 0x7fc5c0137ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:28.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c608c640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5c00a40f0 0x7fc5c013b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:53:28.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c588b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c006eeb0 0x7fc5c0137ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:36892/0 (socket says 172.21.15.100:36892) 2024-09-13T11:53:28.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c588b640 1 -- 172.21.15.100:0/3452654434 learned_addr learned my addr 172.21.15.100:0/3452654434 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:28.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c588b640 10 monclient(hunting): get_auth_request con 0x7fc5c006eeb0 auth_method 0 2024-09-13T11:53:28.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c588b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:28.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c588b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:28.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c588b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:28.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c588b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:28.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c588b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:28.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c588b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:28.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c588b640 10 monclient(hunting): handle_auth_done global_id 4436 payload 824 2024-09-13T11:53:28.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c508a640 1 --2- 172.21.15.100:0/3452654434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5c00a37d0 0x7fc5c01380e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:28.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c588b640 1 -- 172.21.15.100:0/3452654434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5c00a37d0 msgr2=0x7fc5c01380e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:28.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c588b640 1 --2- 172.21.15.100:0/3452654434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5c00a37d0 0x7fc5c01380e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c588b640 1 -- 172.21.15.100:0/3452654434 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5c00a40f0 msgr2=0x7fc5c013b560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:53:28.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c588b640 1 --2- 172.21.15.100:0/3452654434 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5c00a40f0 0x7fc5c013b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c588b640 10 monclient: _finish_hunting 0 2024-09-13T11:53:28.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c588b640 1 monclient: found mon.noname-c 2024-09-13T11:53:28.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c588b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:28.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c588b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:53:28.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c588b640 1 -- 172.21.15.100:0/3452654434 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c013f730 con 0x7fc5c006eeb0 2024-09-13T11:53:28.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c508a640 10 monclient: get_auth_request con 0x7fc5c00a37d0 auth_method 0 2024-09-13T11:53:28.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c508a640 1 --2- 172.21.15.100:0/3452654434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5c00a37d0 0x7fc5c01380e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:53:28.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c588b640 1 --2- 172.21.15.100:0/3452654434 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c006eeb0 0x7fc5c0137ba0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc0098a0 tx=0x7fc5bc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:28.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5b67fc640 1 -- 172.21.15.100:0/3452654434 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc041070 con 0x7fc5c006eeb0 2024-09-13T11:53:28.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:28.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5b67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:28.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5b67fc640 10 monclient: dump: 2024-09-13T11:53:28.713 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:28.713 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:28.713 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.713 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.713 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:28.714 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:28.714 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:28.714 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:28.714 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:28.714 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:28.714 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5b67fc640 10 monclient: _finish_auth 0 2024-09-13T11:53:28.714 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5b67fc640 10 monclient: _check_auth_tickets 2024-09-13T11:53:28.714 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:28.714 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c7aef640 5 monclient: authenticate success, global_id 4436 2024-09-13T11:53:28.714 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5b67fc640 1 -- 172.21.15.100:0/3452654434 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5bc03c670 con 0x7fc5c006eeb0 2024-09-13T11:53:28.714 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5b67fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:28.714 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5b67fc640 1 -- 172.21.15.100:0/3452654434 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc037040 con 0x7fc5c006eeb0 2024-09-13T11:53:28.714 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c7aef640 10 monclient: _renew_subs 2024-09-13T11:53:28.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c7aef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:28.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c7aef640 1 -- 172.21.15.100:0/3452654434 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5c013f280 con 0x7fc5c006eeb0 2024-09-13T11:53:28.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5b7fff640 4 set_mon_vals no callback set 2024-09-13T11:53:28.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:28.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:28.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5b67fc640 10 monclient: dump: 2024-09-13T11:53:28.715 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:28.715 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:28.715 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.715 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.715 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:28.715 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:28.715 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:28.715 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:28.716 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:28.716 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:28.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.693+0000 7fc5c7aef640 10 monclient: _renew_subs 2024-09-13T11:53:28.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.694+0000 7fc5c7aef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:28.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.694+0000 7fc5c7aef640 1 -- 172.21.15.100:0/3452654434 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5c0073280 con 0x7fc5c006eeb0 2024-09-13T11:53:28.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.694+0000 7fc5c7aef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:28.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.694+0000 7fc5c7aef640 1 --2- 172.21.15.100:0/3452654434 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c013ec20 0x7fc5c013baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:28.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.694+0000 7fc5c7aef640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:28.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.694+0000 7fc5c7aef640 1 -- 172.21.15.100:0/3452654434 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fc5c006f330 con 0x7fc5c013ec20 2024-09-13T11:53:28.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.694+0000 7fc5c508a640 1 --2- 172.21.15.100:0/3452654434 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c013ec20 0x7fc5c013baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:28.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.694+0000 7fc5c508a640 10 monclient: get_auth_request con 0x7fc5c013ec20 auth_method 0 2024-09-13T11:53:28.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.694+0000 7fc5c508a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:28.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.694+0000 7fc5c508a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:28.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.694+0000 7fc5c508a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:28.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.694+0000 7fc5b67fc640 1 -- 172.21.15.100:0/3452654434 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fc5bc044070 con 0x7fc5c006eeb0 2024-09-13T11:53:28.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.694+0000 7fc5c508a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:28.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.694+0000 7fc5c508a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:28.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.694+0000 7fc5c508a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:28.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.695+0000 7fc5c508a640 10 monclient(hunting): handle_auth_done global_id 4439 payload 824 2024-09-13T11:53:28.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.695+0000 7fc5c508a640 1 -- 172.21.15.100:0/3452654434 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc5c00a3bb0 con 0x7fc5c013ec20 2024-09-13T11:53:28.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.695+0000 7fc5b67fc640 1 --2- 172.21.15.100:0/3452654434 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc5a403df80 0x7fc5a4040440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:28.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.695+0000 7fc5b67fc640 1 -- 172.21.15.100:0/3452654434 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 4738+0+0 (secure 0 0 0) 0x7fc5bc072f90 con 0x7fc5c006eeb0 2024-09-13T11:53:28.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.695+0000 7fc5c508a640 1 --2- 172.21.15.100:0/3452654434 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c013ec20 0x7fc5c013baa0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0140880 tx=0x7fc5b000d950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:28.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.695+0000 7fc5b67fc640 1 -- 172.21.15.100:0/3452654434 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0016070 con 0x7fc5c013ec20 2024-09-13T11:53:28.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.695+0000 7fc5b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:28.718 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.695+0000 7fc5b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:28.718 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.695+0000 7fc5b67fc640 10 monclient: dump: 2024-09-13T11:53:28.718 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:28.718 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:28.718 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.718 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.718 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:28.718 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:28.718 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:28.718 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:28.718 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:28.718 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:28.718 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.695+0000 7fc5c608c640 1 --2- 172.21.15.100:0/3452654434 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc5a403df80 0x7fc5a4040440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:28.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.695+0000 7fc5c608c640 10 monclient: get_auth_request con 0x7fc5a403df80 auth_method 0 2024-09-13T11:53:28.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.695+0000 7fc5b67fc640 1 -- 172.21.15.100:0/3452654434 <== mon.2 v2:172.21.15.100:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc5b0012800 con 0x7fc5c013ec20 2024-09-13T11:53:28.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.695+0000 7fc5b67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:28.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.695+0000 7fc5b67fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:28.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.695+0000 7fc5b67fc640 1 -- 172.21.15.100:0/3452654434 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c013ec20 msgr2=0x7fc5c013baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:28.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.695+0000 7fc5b67fc640 1 --2- 172.21.15.100:0/3452654434 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c013ec20 0x7fc5c013baa0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0140880 tx=0x7fc5b000d950 comp rx=0 tx=0).stop 2024-09-13T11:53:28.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.695+0000 7fc5b67fc640 1 -- 172.21.15.100:0/3452654434 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c013ec20 msgr2=0x7fc5c013baa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:53:28.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.695+0000 7fc5b67fc640 1 --2- 172.21.15.100:0/3452654434 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c013ec20 0x7fc5c013baa0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.695+0000 7fc5c608c640 1 --2- 172.21.15.100:0/3452654434 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc5a403df80 0x7fc5a4040440 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8003cb0 tx=0x7fc5b80091f0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:28.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.701+0000 7fc5c7aef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:53:28.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.701+0000 7fc5c7aef640 1 --2- 172.21.15.100:0/3452654434 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c013bfe0 0x7fc5c0070f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:28.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.701+0000 7fc5c7aef640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:28.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.701+0000 7fc5c7aef640 1 -- 172.21.15.100:0/3452654434 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fc5c013cfb0 con 0x7fc5c013bfe0 2024-09-13T11:53:28.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.701+0000 7fc5c508a640 1 --2- 172.21.15.100:0/3452654434 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c013bfe0 0x7fc5c0070f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:28.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.701+0000 7fc5c508a640 10 monclient: get_auth_request con 0x7fc5c013bfe0 auth_method 0 2024-09-13T11:53:28.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.701+0000 7fc5c508a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:28.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.701+0000 7fc5c508a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:28.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.701+0000 7fc5c508a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:28.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.701+0000 7fc5c508a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:28.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.701+0000 7fc5c508a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:28.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.701+0000 7fc5c508a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:28.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.701+0000 7fc5c508a640 10 monclient(hunting): handle_auth_done global_id 4442 payload 824 2024-09-13T11:53:28.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.701+0000 7fc5c508a640 1 -- 172.21.15.100:0/3452654434 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc5c013c3c0 con 0x7fc5c013bfe0 2024-09-13T11:53:28.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.702+0000 7fc5c508a640 1 --2- 172.21.15.100:0/3452654434 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c013bfe0 0x7fc5c0070f70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0018000 tx=0x7fc5b0003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:28.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.702+0000 7fc5b67fc640 1 -- 172.21.15.100:0/3452654434 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0006b80 con 0x7fc5c013bfe0 2024-09-13T11:53:28.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.702+0000 7fc5b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:28.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.702+0000 7fc5b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:28.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.702+0000 7fc5b67fc640 10 monclient: dump: 2024-09-13T11:53:28.723 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:28.723 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:28.723 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.723 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.723 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:28.723 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:28.723 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:28.723 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:28.723 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:28.723 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:28.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.702+0000 7fc5b67fc640 1 -- 172.21.15.100:0/3452654434 <== mon.2 v2:172.21.15.100:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7fc5b0006b80 con 0x7fc5c013bfe0 2024-09-13T11:53:28.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.702+0000 7fc5b67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:53:28.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.702+0000 7fc5b67fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:28.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.702+0000 7fc5b67fc640 1 -- 172.21.15.100:0/3452654434 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c013bfe0 msgr2=0x7fc5c0070f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:28.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.702+0000 7fc5b67fc640 1 --2- 172.21.15.100:0/3452654434 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c013bfe0 0x7fc5c0070f70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0018000 tx=0x7fc5b0003760 comp rx=0 tx=0).stop 2024-09-13T11:53:28.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.702+0000 7fc5b67fc640 1 -- 172.21.15.100:0/3452654434 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c013bfe0 msgr2=0x7fc5c0070f70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:53:28.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.702+0000 7fc5b67fc640 1 --2- 172.21.15.100:0/3452654434 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c013bfe0 0x7fc5c0070f70 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0018000 tx=0x7fc5b0003760 comp rx=0 tx=0).stop 2024-09-13T11:53:28.724 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:53:28.724 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:53:28.724 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:53:28.724 INFO:teuthology.orchestra.run.smithi100.stdout: "state": "peon", 2024-09-13T11:53:28.724 INFO:teuthology.orchestra.run.smithi100.stdout: "election_epoch": 12, 2024-09-13T11:53:28.725 INFO:teuthology.orchestra.run.smithi100.stdout: "uptime": 47772, 2024-09-13T11:53:28.725 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum": [ 2024-09-13T11:53:28.725 INFO:teuthology.orchestra.run.smithi100.stdout: 1, 2024-09-13T11:53:28.725 INFO:teuthology.orchestra.run.smithi100.stdout: 2 2024-09-13T11:53:28.725 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:28.725 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_age": 0, 2024-09-13T11:53:28.725 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:53:28.725 INFO:teuthology.orchestra.run.smithi100.stdout: "required_con": "2450521708007866372", 2024-09-13T11:53:28.725 INFO:teuthology.orchestra.run.smithi100.stdout: "required_mon": [ 2024-09-13T11:53:28.725 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:53:28.725 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:53:28.725 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:53:28.726 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:53:28.726 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:53:28.726 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:53:28.726 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:53:28.726 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:53:28.726 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:53:28.726 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:53:28.726 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:53:28.726 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:28.726 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:53:28.726 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_mon": [ 2024-09-13T11:53:28.726 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:53:28.726 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:53:28.727 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:53:28.727 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:53:28.727 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:53:28.727 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:53:28.727 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:53:28.727 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:53:28.727 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:53:28.727 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:53:28.727 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:53:28.727 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:28.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:28.727 INFO:teuthology.orchestra.run.smithi100.stdout: "outside_quorum": [], 2024-09-13T11:53:28.727 INFO:teuthology.orchestra.run.smithi100.stdout: "extra_probe_peers": [], 2024-09-13T11:53:28.728 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_provider": [], 2024-09-13T11:53:28.728 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": { 2024-09-13T11:53:28.728 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-09-13T11:53:28.728 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "0b1f4b4d-fad4-4dd3-86ac-a286d80846c0", 2024-09-13T11:53:28.728 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:53:28.728 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:53:28.728 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release": 19, 2024-09-13T11:53:28.728 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release_name": "squid", 2024-09-13T11:53:28.728 INFO:teuthology.orchestra.run.smithi100.stdout: "election_strategy": 1, 2024-09-13T11:53:28.728 INFO:teuthology.orchestra.run.smithi100.stdout: "disallowed_leaders: ": "", 2024-09-13T11:53:28.728 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false, 2024-09-13T11:53:28.728 INFO:teuthology.orchestra.run.smithi100.stdout: "tiebreaker_mon": "", 2024-09-13T11:53:28.728 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_ranks: ": "", 2024-09-13T11:53:28.729 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:53:28.729 INFO:teuthology.orchestra.run.smithi100.stdout: "persistent": [ 2024-09-13T11:53:28.729 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:53:28.729 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:53:28.729 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:53:28.729 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:53:28.729 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:53:28.729 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:53:28.729 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:53:28.729 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:53:28.729 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:53:28.729 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:53:28.729 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:53:28.730 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:28.730 INFO:teuthology.orchestra.run.smithi100.stdout: "optional": [] 2024-09-13T11:53:28.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:28.730 INFO:teuthology.orchestra.run.smithi100.stdout: "mons": [ 2024-09-13T11:53:28.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.730 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:53:28.730 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:53:28.730 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:53:28.730 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:53:28.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.730 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:53:28.730 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3300", 2024-09-13T11:53:28.730 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:28.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:28.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.731 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:53:28.731 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789", 2024-09-13T11:53:28.731 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:28.731 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:28.731 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:28.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:28.731 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789/0", 2024-09-13T11:53:28.731 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-13T11:53:28.731 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:53:28.731 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:53:28.732 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:53:28.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:28.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.732 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:53:28.732 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:53:28.732 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:53:28.732 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:53:28.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.732 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:53:28.732 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:3300", 2024-09-13T11:53:28.732 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:28.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:28.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.733 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:53:28.733 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789", 2024-09-13T11:53:28.733 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:28.733 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:28.733 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:28.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:28.733 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789/0", 2024-09-13T11:53:28.733 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.122:6789/0", 2024-09-13T11:53:28.733 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:53:28.733 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:53:28.733 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:53:28.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:28.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.734 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:53:28.734 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:53:28.734 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:53:28.734 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:53:28.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.734 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:53:28.734 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3301", 2024-09-13T11:53:28.734 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:28.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:28.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.734 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:53:28.735 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790", 2024-09-13T11:53:28.735 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:28.735 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:28.735 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:28.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:28.735 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790/0", 2024-09-13T11:53:28.735 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-13T11:53:28.735 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:53:28.735 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:53:28.735 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:53:28.735 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:28.735 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:28.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:28.736 INFO:teuthology.orchestra.run.smithi100.stdout: "feature_map": { 2024-09-13T11:53:28.736 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": [ 2024-09-13T11:53:28.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.736 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:53:28.736 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:53:28.736 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:53:28.736 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:28.736 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:28.736 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": [ 2024-09-13T11:53:28.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.736 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:53:28.736 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:53:28.736 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 3 2024-09-13T11:53:28.737 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:28.737 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:28.737 INFO:teuthology.orchestra.run.smithi100.stdout: "client": [ 2024-09-13T11:53:28.737 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.737 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:53:28.737 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:53:28.740 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 3 2024-09-13T11:53:28.740 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:28.741 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:28.741 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:28.741 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false 2024-09-13T11:53:28.741 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:53:28.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.702+0000 7fc5c7aef640 1 -- 172.21.15.100:0/3452654434 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc5a403df80 msgr2=0x7fc5a4040440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:28.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.702+0000 7fc5c7aef640 1 --2- 172.21.15.100:0/3452654434 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc5a403df80 0x7fc5a4040440 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8003cb0 tx=0x7fc5b80091f0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.702+0000 7fc5c7aef640 10 monclient: shutdown 2024-09-13T11:53:28.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.702+0000 7fc5c7aef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:28.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.702+0000 7fc5c7aef640 1 -- 172.21.15.100:0/3452654434 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c006eeb0 msgr2=0x7fc5c0137ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:28.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.702+0000 7fc5c7aef640 1 --2- 172.21.15.100:0/3452654434 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5c006eeb0 0x7fc5c0137ba0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc0098a0 tx=0x7fc5bc0046c0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.702+0000 7fc5c608c640 1 -- 172.21.15.100:0/3452654434 reap_dead start 2024-09-13T11:53:28.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.702+0000 7fc5c608c640 1 -- 172.21.15.100:0/3452654434 reap_dead start 2024-09-13T11:53:28.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.702+0000 7fc5c7aef640 1 -- 172.21.15.100:0/3452654434 shutdown_connections 2024-09-13T11:53:28.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.702+0000 7fc5c7aef640 1 -- 172.21.15.100:0/3452654434 >> 172.21.15.100:0/3452654434 conn(0x7fc5c00a83f0 msgr2=0x7fc5c009da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:28.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.702+0000 7fc5c7aef640 1 -- 172.21.15.100:0/3452654434 shutdown_connections 2024-09-13T11:53:28.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.702+0000 7fc5c7aef640 1 -- 172.21.15.100:0/3452654434 wait complete. 2024-09-13T11:53:28.744 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-13T11:53:28.744 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-13T11:53:28.744 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-13T11:53:28.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.841+0000 7fb425dcd640 10 monclient: get_monmap_and_config 2024-09-13T11:53:28.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.841+0000 7fb425dcd640 10 monclient: build_initial_monmap 2024-09-13T11:53:28.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.841+0000 7fb425dcd640 10 monclient: monmap: 2024-09-13T11:53:28.843 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:28.843 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:28.843 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:53:28.842636+0000 2024-09-13T11:53:28.843 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:53:28.842636+0000 2024-09-13T11:53:28.843 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:28.843 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:28.844 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:53:28.844 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:53:28.844 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:53:28.844 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:53:28.844 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:53:28.844 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:28.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.842+0000 7fb425dcd640 1 Processor -- start 2024-09-13T11:53:28.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.842+0000 7fb425dcd640 1 -- start start 2024-09-13T11:53:28.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.842+0000 7fb425dcd640 10 monclient: init 2024-09-13T11:53:28.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.842+0000 7fb425dcd640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:28.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb425dcd640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-13T11:53:28.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb425dcd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb4200a37d0 0x7fb4200a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:28.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb425dcd640 10 monclient(hunting): picked mon.noname-c con 0x7fb4200a37d0 addr v2:172.21.15.122:3300/0 2024-09-13T11:53:28.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb425dcd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb42006eeb0 0x7fb4200a40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:28.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb425dcd640 10 monclient(hunting): picked mon.noname-b con 0x7fb42006eeb0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:28.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb425dcd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb420137970 addr v1:172.21.15.122:6789/0 2024-09-13T11:53:28.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb425dcd640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb42006e950 con 0x7fb420137970 2024-09-13T11:53:28.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb425dcd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:28.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb425dcd640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7fb42009fde0 con 0x7fb4200a37d0 2024-09-13T11:53:28.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb425dcd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:28.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb425dcd640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fb42006f330 con 0x7fb42006eeb0 2024-09-13T11:53:28.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb425dcd640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:28.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb425dcd640 10 monclient(hunting): authenticate will time out at 789.436108s 2024-09-13T11:53:28.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb41effd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb42006eeb0 0x7fb4200a40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:28.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb41effd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb42006eeb0 0x7fb4200a40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:36920/0 (socket says 172.21.15.100:36920) 2024-09-13T11:53:28.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb41effd640 1 -- 172.21.15.100:0/1693677044 learned_addr learned my addr 172.21.15.100:0/1693677044 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:28.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb41effd640 10 monclient(hunting): get_auth_request con 0x7fb42006eeb0 auth_method 0 2024-09-13T11:53:28.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb41effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:28.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb41effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:28.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb41effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:28.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb41effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:28.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb41effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:28.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb41effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:28.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb41f7fe640 1 --2- 172.21.15.100:0/1693677044 >> v2:172.21.15.122:3300/0 conn(0x7fb4200a37d0 0x7fb4200a3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:28.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb41effd640 10 monclient(hunting): handle_auth_done global_id 4445 payload 274 2024-09-13T11:53:28.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb41effd640 1 -- 172.21.15.100:0/1693677044 >> v2:172.21.15.122:3300/0 conn(0x7fb4200a37d0 msgr2=0x7fb4200a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:28.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb41effd640 1 --2- 172.21.15.100:0/1693677044 >> v2:172.21.15.122:3300/0 conn(0x7fb4200a37d0 0x7fb4200a3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb41effd640 1 -- 172.21.15.100:0/1693677044 >> v1:172.21.15.122:6789/0 conn(0x7fb420137970 legacy=0x7fb420137e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:53:28.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb41effd640 10 monclient: _finish_hunting 0 2024-09-13T11:53:28.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb41effd640 1 monclient: found mon.noname-b 2024-09-13T11:53:28.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb41effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:28.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb41effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:53:28.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.843+0000 7fb41effd640 1 -- 172.21.15.100:0/1693677044 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4200a5240 con 0x7fb42006eeb0 2024-09-13T11:53:28.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb41effd640 1 --2- 172.21.15.100:0/1693677044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb42006eeb0 0x7fb4200a40f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb414009870 tx=0x7fb41402a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a84af65b01d5958 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:28.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb41e7fc640 1 -- 172.21.15.100:0/1693677044 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb414037070 con 0x7fb42006eeb0 2024-09-13T11:53:28.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb41e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:28.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb41e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:53:28.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb41e7fc640 10 monclient: dump: 2024-09-13T11:53:28.860 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:28.860 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:28.860 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.860 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.860 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:28.861 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:28.861 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:28.861 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:28.861 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:28.861 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:28.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb41e7fc640 10 monclient: _finish_auth 0 2024-09-13T11:53:28.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb41e7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:53:28.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb41e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:28.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb41e7fc640 1 -- 172.21.15.100:0/1693677044 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb41402adf0 con 0x7fb42006eeb0 2024-09-13T11:53:28.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb425dcd640 5 monclient: authenticate success, global_id 4445 2024-09-13T11:53:28.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb425dcd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:53:28.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb41e7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:28.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb425dcd640 10 monclient: get_monmap_and_config success 2024-09-13T11:53:28.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb41e7fc640 1 -- 172.21.15.100:0/1693677044 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb414032660 con 0x7fb42006eeb0 2024-09-13T11:53:28.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb425dcd640 4 set_mon_vals no callback set 2024-09-13T11:53:28.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb41e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:28.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb41e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:28.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb41e7fc640 10 monclient: dump: 2024-09-13T11:53:28.862 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:28.862 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:28.862 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.862 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.862 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:28.862 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:28.863 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:28.863 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:28.863 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:28.863 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:28.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb425dcd640 10 monclient: shutdown 2024-09-13T11:53:28.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb425dcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:28.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb425dcd640 1 -- 172.21.15.100:0/1693677044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb42006eeb0 msgr2=0x7fb4200a40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:28.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb425dcd640 1 --2- 172.21.15.100:0/1693677044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb42006eeb0 0x7fb4200a40f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb414009870 tx=0x7fb41402a0a0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb425dcd640 1 -- 172.21.15.100:0/1693677044 shutdown_connections 2024-09-13T11:53:28.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb425dcd640 1 --2- 172.21.15.100:0/1693677044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb42006eeb0 0x7fb4200a40f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb425dcd640 1 --2- 172.21.15.100:0/1693677044 >> v2:172.21.15.122:3300/0 conn(0x7fb4200a37d0 0x7fb4200a3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb425dcd640 1 -- 172.21.15.100:0/1693677044 >> 172.21.15.100:0/1693677044 conn(0x7fb4200a83f0 msgr2=0x7fb4200a48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:28.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb425dcd640 1 -- 172.21.15.100:0/1693677044 shutdown_connections 2024-09-13T11:53:28.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.844+0000 7fb425dcd640 1 -- 172.21.15.100:0/1693677044 wait complete. 2024-09-13T11:53:28.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.845+0000 7fb425dcd640 10 monclient: build_initial_monmap 2024-09-13T11:53:28.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.845+0000 7fb425dcd640 10 monclient: monmap: 2024-09-13T11:53:28.864 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:53:28.864 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:53:28.864 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:53:28.864 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:53:28.864 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:53:28.864 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:28.864 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:53:28.865 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:53:28.865 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:53:28.865 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:28.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.845+0000 7fb425dcd640 1 Processor -- start 2024-09-13T11:53:28.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb425dcd640 1 -- start start 2024-09-13T11:53:28.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb425dcd640 10 monclient: init 2024-09-13T11:53:28.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb425dcd640 10 monclient: _reopen_session rank -1 2024-09-13T11:53:28.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb425dcd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:53:28.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb425dcd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb42006eeb0 0x7fb4201ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:28.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb425dcd640 10 monclient(hunting): picked mon.noname-b con 0x7fb42006eeb0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:53:28.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb425dcd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4200a37d0 0x7fb4201cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:28.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb425dcd640 10 monclient(hunting): picked mon.noname-c con 0x7fb4200a37d0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:53:28.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb425dcd640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb420137970 0x7fb420124280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:28.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb425dcd640 10 monclient(hunting): picked mon.noname-a con 0x7fb420137970 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:53:28.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb425dcd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:28.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb425dcd640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fb420138530 con 0x7fb420137970 2024-09-13T11:53:28.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb425dcd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:28.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb425dcd640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb4200a4630 con 0x7fb42006eeb0 2024-09-13T11:53:28.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb425dcd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:28.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb425dcd640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fb4201386b0 con 0x7fb4200a37d0 2024-09-13T11:53:28.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb425dcd640 10 monclient(hunting): _renew_subs 2024-09-13T11:53:28.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb425dcd640 10 monclient(hunting): authenticate will time out at 789.439459s 2024-09-13T11:53:28.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41ffff640 1 -- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb420137970 msgr2=0x7fb420124280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:3300/0 2024-09-13T11:53:28.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41ffff640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb420137970 0x7fb420124280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:53:28.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41effd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4200a37d0 0x7fb4201cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:28.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41effd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4200a37d0 0x7fb4201cd1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:36922/0 (socket says 172.21.15.100:36922) 2024-09-13T11:53:28.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41effd640 1 -- 172.21.15.100:0/2703972905 learned_addr learned my addr 172.21.15.100:0/2703972905 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:53:28.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41effd640 10 monclient(hunting): get_auth_request con 0x7fb4200a37d0 auth_method 0 2024-09-13T11:53:28.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:28.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:28.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:28.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41f7fe640 1 --2- 172.21.15.100:0/2703972905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb42006eeb0 0x7fb4201ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:28.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:28.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:28.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:28.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41f7fe640 10 monclient(hunting): get_auth_request con 0x7fb42006eeb0 auth_method 0 2024-09-13T11:53:28.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:28.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:28.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:28.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:28.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:28.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:28.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41effd640 10 monclient(hunting): handle_auth_done global_id 4448 payload 824 2024-09-13T11:53:28.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41effd640 1 -- 172.21.15.100:0/2703972905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb42006eeb0 msgr2=0x7fb4201ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:28.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41effd640 1 --2- 172.21.15.100:0/2703972905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb42006eeb0 0x7fb4201ccc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41effd640 1 -- 172.21.15.100:0/2703972905 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb420137970 msgr2=0x7fb420124280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:53:28.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41effd640 1 --2- 172.21.15.100:0/2703972905 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb420137970 0x7fb420124280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41effd640 10 monclient: _finish_hunting 0 2024-09-13T11:53:28.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41effd640 1 monclient: found mon.noname-c 2024-09-13T11:53:28.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:53:28.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:53:28.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41effd640 1 -- 172.21.15.100:0/2703972905 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4201375d0 con 0x7fb4200a37d0 2024-09-13T11:53:28.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.846+0000 7fb41effd640 1 --2- 172.21.15.100:0/2703972905 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4200a37d0 0x7fb4201cd1b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb414000c00 tx=0x7fb4140046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:28.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.847+0000 7fb407fff640 1 -- 172.21.15.100:0/2703972905 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb414041020 con 0x7fb4200a37d0 2024-09-13T11:53:28.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.847+0000 7fb407fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:28.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.847+0000 7fb407fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:53:28.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.847+0000 7fb407fff640 10 monclient: dump: 2024-09-13T11:53:28.870 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:28.870 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:28.870 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.870 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.870 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:28.870 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:28.870 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:28.870 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:28.870 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:28.871 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:28.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.847+0000 7fb407fff640 10 monclient: _finish_auth 0 2024-09-13T11:53:28.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.847+0000 7fb407fff640 10 monclient: _check_auth_tickets 2024-09-13T11:53:28.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.847+0000 7fb407fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:53:28.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.847+0000 7fb407fff640 1 -- 172.21.15.100:0/2703972905 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb41403b620 con 0x7fb4200a37d0 2024-09-13T11:53:28.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.847+0000 7fb425dcd640 5 monclient: authenticate success, global_id 4448 2024-09-13T11:53:28.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.847+0000 7fb407fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:53:28.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.847+0000 7fb425dcd640 10 monclient: _renew_subs 2024-09-13T11:53:28.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.847+0000 7fb407fff640 1 -- 172.21.15.100:0/2703972905 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb414037040 con 0x7fb4200a37d0 2024-09-13T11:53:28.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.847+0000 7fb425dcd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:28.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.847+0000 7fb425dcd640 1 -- 172.21.15.100:0/2703972905 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb420128010 con 0x7fb4200a37d0 2024-09-13T11:53:28.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.847+0000 7fb407fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:28.872 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.847+0000 7fb407fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:53:28.872 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.847+0000 7fb407fff640 10 monclient: dump: 2024-09-13T11:53:28.872 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:28.872 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:28.872 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.872 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.872 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:28.872 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:28.872 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:28.872 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:28.872 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:28.873 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:28.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.847+0000 7fb425dcd640 10 monclient: _renew_subs 2024-09-13T11:53:28.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.847+0000 7fb41dffb640 4 set_mon_vals no callback set 2024-09-13T11:53:28.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.847+0000 7fb425dcd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:53:28.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.847+0000 7fb425dcd640 1 -- 172.21.15.100:0/2703972905 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4200a1820 con 0x7fb4200a37d0 2024-09-13T11:53:28.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.848+0000 7fb407fff640 1 -- 172.21.15.100:0/2703972905 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fb414044020 con 0x7fb4200a37d0 2024-09-13T11:53:28.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.848+0000 7fb425dcd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:28.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.848+0000 7fb425dcd640 1 --2- 172.21.15.100:0/2703972905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb420124f80 0x7fb42002d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:28.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.848+0000 7fb425dcd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:28.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.848+0000 7fb425dcd640 1 -- 172.21.15.100:0/2703972905 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb420137e20 con 0x7fb420124f80 2024-09-13T11:53:28.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.848+0000 7fb407fff640 1 --2- 172.21.15.100:0/2703972905 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fb40c03df50 0x7fb40c040410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:28.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.848+0000 7fb407fff640 1 -- 172.21.15.100:0/2703972905 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 4738+0+0 (secure 0 0 0) 0x7fb414072970 con 0x7fb4200a37d0 2024-09-13T11:53:28.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.848+0000 7fb41ffff640 1 --2- 172.21.15.100:0/2703972905 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fb40c03df50 0x7fb40c040410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:28.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.848+0000 7fb41f7fe640 1 --2- 172.21.15.100:0/2703972905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb420124f80 0x7fb42002d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:28.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.848+0000 7fb41ffff640 10 monclient: get_auth_request con 0x7fb40c03df50 auth_method 0 2024-09-13T11:53:28.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.848+0000 7fb41f7fe640 10 monclient: get_auth_request con 0x7fb420124f80 auth_method 0 2024-09-13T11:53:28.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.848+0000 7fb41f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:28.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.848+0000 7fb41f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:28.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.848+0000 7fb41f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:28.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.848+0000 7fb41f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:28.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.848+0000 7fb41f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:28.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.848+0000 7fb41f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:28.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.848+0000 7fb41f7fe640 10 monclient(hunting): handle_auth_done global_id 4426 payload 824 2024-09-13T11:53:28.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.848+0000 7fb41f7fe640 1 -- 172.21.15.100:0/2703972905 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb420072610 con 0x7fb420124f80 2024-09-13T11:53:28.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.848+0000 7fb41ffff640 1 --2- 172.21.15.100:0/2703972905 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fb40c03df50 0x7fb40c040410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb420126070 tx=0x7fb418007d10 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:28.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.849+0000 7fb41f7fe640 1 --2- 172.21.15.100:0/2703972905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb420124f80 0x7fb42002d1f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb41000f7e0 tx=0x7fb41000fca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:28.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.849+0000 7fb407fff640 1 -- 172.21.15.100:0/2703972905 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41000dc20 con 0x7fb420124f80 2024-09-13T11:53:28.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.849+0000 7fb407fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:28.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.849+0000 7fb407fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:53:28.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.849+0000 7fb407fff640 10 monclient: dump: 2024-09-13T11:53:28.875 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:28.875 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:28.875 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.875 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.876 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:28.876 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:28.876 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:28.876 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:28.876 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:28.876 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:28.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.849+0000 7fb407fff640 1 -- 172.21.15.100:0/2703972905 <== mon.1 v2:172.21.15.122:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb41000dc20 con 0x7fb420124f80 2024-09-13T11:53:28.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.849+0000 7fb407fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:53:28.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.849+0000 7fb407fff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:53:28.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.849+0000 7fb407fff640 1 -- 172.21.15.100:0/2703972905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb420124f80 msgr2=0x7fb42002d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:28.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.849+0000 7fb407fff640 1 --2- 172.21.15.100:0/2703972905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb420124f80 0x7fb42002d1f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb41000f7e0 tx=0x7fb41000fca0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.849+0000 7fb407fff640 1 -- 172.21.15.100:0/2703972905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb420124f80 msgr2=0x7fb42002d1f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:53:28.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.849+0000 7fb407fff640 1 --2- 172.21.15.100:0/2703972905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb420124f80 0x7fb42002d1f0 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb41000f7e0 tx=0x7fb41000fca0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.855+0000 7fb425dcd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:53:28.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.855+0000 7fb425dcd640 1 --2- 172.21.15.100:0/2703972905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb420125360 0x7fb42002d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:53:28.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.855+0000 7fb425dcd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:53:28.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.855+0000 7fb425dcd640 1 -- 172.21.15.100:0/2703972905 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb420126810 con 0x7fb420125360 2024-09-13T11:53:28.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.856+0000 7fb41f7fe640 1 --2- 172.21.15.100:0/2703972905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb420125360 0x7fb42002d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:53:28.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.856+0000 7fb41f7fe640 10 monclient: get_auth_request con 0x7fb420125360 auth_method 0 2024-09-13T11:53:28.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.856+0000 7fb41f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:53:28.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.856+0000 7fb41f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:53:28.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.856+0000 7fb41f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:53:28.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.856+0000 7fb41f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:53:28.880 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.856+0000 7fb41f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:53:28.880 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.856+0000 7fb41f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:53:28.880 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.856+0000 7fb41f7fe640 10 monclient(hunting): handle_auth_done global_id 4429 payload 824 2024-09-13T11:53:28.880 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.856+0000 7fb41f7fe640 1 -- 172.21.15.100:0/2703972905 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb42006ca60 con 0x7fb420125360 2024-09-13T11:53:28.880 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.856+0000 7fb41f7fe640 1 --2- 172.21.15.100:0/2703972905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb420125360 0x7fb42002d730 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb41000f7e0 tx=0x7fb4100046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:53:28.880 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.856+0000 7fb407fff640 1 -- 172.21.15.100:0/2703972905 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb410029020 con 0x7fb420125360 2024-09-13T11:53:28.880 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.856+0000 7fb407fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:53:28.880 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.856+0000 7fb407fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:53:28.880 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.856+0000 7fb407fff640 10 monclient: dump: 2024-09-13T11:53:28.880 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:53:28.880 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:53:28.881 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.881 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:53:28.881 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:53:28.881 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:53:28.881 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:53:28.881 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:53:28.881 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:53:28.881 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:53:28.881 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.856+0000 7fb407fff640 1 -- 172.21.15.100:0/2703972905 <== mon.1 v2:172.21.15.122:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4877 (secure 0 0 0) 0x7fb410003e30 con 0x7fb420125360 2024-09-13T11:53:28.881 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.856+0000 7fb407fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:53:28.881 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.856+0000 7fb407fff640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:53:28.881 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.856+0000 7fb407fff640 1 -- 172.21.15.100:0/2703972905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb420125360 msgr2=0x7fb42002d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:28.882 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.856+0000 7fb407fff640 1 --2- 172.21.15.100:0/2703972905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb420125360 0x7fb42002d730 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb41000f7e0 tx=0x7fb4100046c0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.882 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.856+0000 7fb407fff640 1 -- 172.21.15.100:0/2703972905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb420125360 msgr2=0x7fb42002d730 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:53:28.882 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.856+0000 7fb407fff640 1 --2- 172.21.15.100:0/2703972905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb420125360 0x7fb42002d730 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb41000f7e0 tx=0x7fb4100046c0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.882 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:53:28.882 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:53:28.882 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:53:28.882 INFO:teuthology.orchestra.run.smithi100.stdout: "state": "leader", 2024-09-13T11:53:28.882 INFO:teuthology.orchestra.run.smithi100.stdout: "election_epoch": 12, 2024-09-13T11:53:28.883 INFO:teuthology.orchestra.run.smithi100.stdout: "uptime": 47886, 2024-09-13T11:53:28.883 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum": [ 2024-09-13T11:53:28.883 INFO:teuthology.orchestra.run.smithi100.stdout: 1, 2024-09-13T11:53:28.883 INFO:teuthology.orchestra.run.smithi100.stdout: 2 2024-09-13T11:53:28.883 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:28.883 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_age": 0, 2024-09-13T11:53:28.883 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:53:28.883 INFO:teuthology.orchestra.run.smithi100.stdout: "required_con": "2450521708007866372", 2024-09-13T11:53:28.883 INFO:teuthology.orchestra.run.smithi100.stdout: "required_mon": [ 2024-09-13T11:53:28.883 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:53:28.883 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:53:28.883 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:53:28.884 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:53:28.884 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:53:28.884 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:53:28.884 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:53:28.884 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:53:28.884 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:53:28.884 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:53:28.884 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:53:28.884 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:28.884 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:53:28.884 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_mon": [ 2024-09-13T11:53:28.885 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:53:28.885 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:53:28.885 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:53:28.885 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:53:28.885 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:53:28.885 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:53:28.885 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:53:28.885 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:53:28.885 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:53:28.885 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:53:28.885 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:53:28.885 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:28.886 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:28.886 INFO:teuthology.orchestra.run.smithi100.stdout: "outside_quorum": [], 2024-09-13T11:53:28.886 INFO:teuthology.orchestra.run.smithi100.stdout: "extra_probe_peers": [], 2024-09-13T11:53:28.886 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_provider": [], 2024-09-13T11:53:28.886 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": { 2024-09-13T11:53:28.886 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-09-13T11:53:28.886 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "0b1f4b4d-fad4-4dd3-86ac-a286d80846c0", 2024-09-13T11:53:28.886 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:53:28.886 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:53:28.886 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release": 19, 2024-09-13T11:53:28.886 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release_name": "squid", 2024-09-13T11:53:28.886 INFO:teuthology.orchestra.run.smithi100.stdout: "election_strategy": 1, 2024-09-13T11:53:28.887 INFO:teuthology.orchestra.run.smithi100.stdout: "disallowed_leaders: ": "", 2024-09-13T11:53:28.887 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false, 2024-09-13T11:53:28.887 INFO:teuthology.orchestra.run.smithi100.stdout: "tiebreaker_mon": "", 2024-09-13T11:53:28.887 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_ranks: ": "", 2024-09-13T11:53:28.887 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:53:28.887 INFO:teuthology.orchestra.run.smithi100.stdout: "persistent": [ 2024-09-13T11:53:28.887 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:53:28.887 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:53:28.887 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:53:28.888 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:53:28.888 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:53:28.888 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:53:28.888 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:53:28.888 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:53:28.888 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:53:28.888 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:53:28.888 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:53:28.888 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:28.888 INFO:teuthology.orchestra.run.smithi100.stdout: "optional": [] 2024-09-13T11:53:28.888 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:28.888 INFO:teuthology.orchestra.run.smithi100.stdout: "mons": [ 2024-09-13T11:53:28.889 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.889 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:53:28.889 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:53:28.889 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:53:28.889 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:53:28.889 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.889 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:53:28.889 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3300", 2024-09-13T11:53:28.889 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:28.889 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:28.889 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.890 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:53:28.890 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789", 2024-09-13T11:53:28.890 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:28.890 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:28.890 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:28.890 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:28.890 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789/0", 2024-09-13T11:53:28.890 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-13T11:53:28.890 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:53:28.890 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:53:28.890 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:53:28.890 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:28.891 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.891 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:53:28.891 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:53:28.891 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:53:28.891 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:53:28.891 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.891 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:53:28.891 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:3300", 2024-09-13T11:53:28.891 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:28.891 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:28.891 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.892 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:53:28.892 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789", 2024-09-13T11:53:28.892 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:28.892 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:28.892 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:28.892 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:28.892 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789/0", 2024-09-13T11:53:28.892 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.122:6789/0", 2024-09-13T11:53:28.892 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:53:28.893 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:53:28.893 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:53:28.893 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:28.893 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.893 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:53:28.893 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:53:28.893 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:53:28.893 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:53:28.893 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.893 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:53:28.893 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3301", 2024-09-13T11:53:28.893 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:28.894 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:28.894 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.894 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:53:28.894 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790", 2024-09-13T11:53:28.894 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:53:28.894 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:28.894 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:28.894 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:28.894 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790/0", 2024-09-13T11:53:28.894 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-13T11:53:28.894 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:53:28.895 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:53:28.895 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:53:28.895 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:28.895 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:28.895 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:28.895 INFO:teuthology.orchestra.run.smithi100.stdout: "feature_map": { 2024-09-13T11:53:28.895 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": [ 2024-09-13T11:53:28.896 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.896 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:53:28.896 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:53:28.896 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:53:28.896 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:28.896 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:28.896 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": [ 2024-09-13T11:53:28.896 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.896 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:53:28.896 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:53:28.896 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 3 2024-09-13T11:53:28.896 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:28.897 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:28.897 INFO:teuthology.orchestra.run.smithi100.stdout: "client": [ 2024-09-13T11:53:28.897 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.897 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:53:28.897 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:53:28.897 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 5 2024-09-13T11:53:28.897 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:28.897 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:53:28.897 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": [ 2024-09-13T11:53:28.897 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:53:28.897 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:53:28.897 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:53:28.898 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:53:28.898 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:53:28.898 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:53:28.898 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:53:28.898 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false 2024-09-13T11:53:28.898 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:53:28.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.857+0000 7fb425dcd640 1 -- 172.21.15.100:0/2703972905 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fb40c03df50 msgr2=0x7fb40c040410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:28.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.857+0000 7fb425dcd640 1 --2- 172.21.15.100:0/2703972905 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fb40c03df50 0x7fb40c040410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb420126070 tx=0x7fb418007d10 comp rx=0 tx=0).stop 2024-09-13T11:53:28.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.857+0000 7fb425dcd640 10 monclient: shutdown 2024-09-13T11:53:28.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.857+0000 7fb425dcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:53:28.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.857+0000 7fb425dcd640 1 -- 172.21.15.100:0/2703972905 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4200a37d0 msgr2=0x7fb4201cd1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:53:28.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.857+0000 7fb425dcd640 1 --2- 172.21.15.100:0/2703972905 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4200a37d0 0x7fb4201cd1b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb414000c00 tx=0x7fb4140046c0 comp rx=0 tx=0).stop 2024-09-13T11:53:28.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.857+0000 7fb41ffff640 1 -- 172.21.15.100:0/2703972905 reap_dead start 2024-09-13T11:53:28.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.857+0000 7fb41ffff640 1 -- 172.21.15.100:0/2703972905 reap_dead start 2024-09-13T11:53:28.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.857+0000 7fb425dcd640 1 -- 172.21.15.100:0/2703972905 shutdown_connections 2024-09-13T11:53:28.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.857+0000 7fb425dcd640 1 -- 172.21.15.100:0/2703972905 >> 172.21.15.100:0/2703972905 conn(0x7fb4200a83f0 msgr2=0x7fb42009da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:53:28.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.857+0000 7fb425dcd640 1 -- 172.21.15.100:0/2703972905 shutdown_connections 2024-09-13T11:53:28.902 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:53:28.857+0000 7fb425dcd640 1 -- 172.21.15.100:0/2703972905 wait complete. 2024-09-13T11:53:28.902 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-13T11:53:28.902 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-13T11:53:29.360 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[34554]: starting. 2024-09-13T11:53:29.360 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[34554]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-34552 2024-09-13T11:53:29.360 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[34554]: shutting down. 2024-09-13T11:53:31.459 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:31.460 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34587]: starting. 2024-09-13T11:53:31.460 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34587]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-34552 2024-09-13T11:53:31.460 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34587]: created object 0... 2024-09-13T11:53:31.460 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34587]: created object 25... 2024-09-13T11:53:31.460 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34587]: created object 49... 2024-09-13T11:53:31.460 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34587]: finishing. 2024-09-13T11:53:31.460 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34587]: shutting down. 2024-09-13T11:53:31.472 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:31.472 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[34589]: starting. 2024-09-13T11:53:31.472 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[34589]: listing objects. 2024-09-13T11:53:31.472 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[34589]: listed object 0... 2024-09-13T11:53:31.472 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[34589]: listed object 25... 2024-09-13T11:53:31.472 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[34589]: saw 50 objects 2024-09-13T11:53:31.472 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[34589]: shutting down. 2024-09-13T11:53:32.379 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:32.379 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[34588]: starting. 2024-09-13T11:53:32.379 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[34588]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-34552 2024-09-13T11:53:32.379 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[34588]: shutting down. 2024-09-13T11:53:32.381 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:32.381 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:32.381 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:53:32.383 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:53:34.478 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34659]: starting. 2024-09-13T11:53:34.479 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34659]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-34658 2024-09-13T11:53:34.479 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34659]: created object 0... 2024-09-13T11:53:34.479 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34659]: created object 25... 2024-09-13T11:53:34.479 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34659]: created object 49... 2024-09-13T11:53:34.479 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34659]: finishing. 2024-09-13T11:53:34.479 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34659]: shutting down. 2024-09-13T11:53:35.394 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[34660]: starting. 2024-09-13T11:53:35.394 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[34660]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-34658 2024-09-13T11:53:35.394 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[34660]: shutting down. 2024-09-13T11:53:37.490 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:37.490 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34695]: starting. 2024-09-13T11:53:37.491 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34695]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-34658 2024-09-13T11:53:37.491 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34695]: created object 0... 2024-09-13T11:53:37.491 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34695]: created object 25... 2024-09-13T11:53:37.491 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34695]: created object 49... 2024-09-13T11:53:37.491 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34695]: finishing. 2024-09-13T11:53:37.491 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34695]: shutting down. 2024-09-13T11:53:37.501 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:37.502 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[34697]: starting. 2024-09-13T11:53:37.502 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[34697]: listing objects. 2024-09-13T11:53:37.502 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[34697]: listed object 0... 2024-09-13T11:53:37.502 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[34697]: listed object 25... 2024-09-13T11:53:37.502 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[34697]: saw 50 objects 2024-09-13T11:53:37.502 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[34697]: shutting down. 2024-09-13T11:53:38.405 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:38.406 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[34696]: starting. 2024-09-13T11:53:38.406 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[34696]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-34658 2024-09-13T11:53:38.406 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[34696]: shutting down. 2024-09-13T11:53:38.407 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:38.407 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:38.407 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:53:38.410 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:53:40.507 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34766]: starting. 2024-09-13T11:53:40.507 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34766]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-34765 2024-09-13T11:53:40.507 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34766]: created object 0... 2024-09-13T11:53:40.507 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34766]: created object 25... 2024-09-13T11:53:40.507 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34766]: created object 49... 2024-09-13T11:53:40.507 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34766]: finishing. 2024-09-13T11:53:40.507 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34766]: shutting down. 2024-09-13T11:53:41.426 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[34767]: starting. 2024-09-13T11:53:41.426 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[34767]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-34765 2024-09-13T11:53:41.427 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[34767]: shutting down. 2024-09-13T11:53:43.841 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:43.841 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34802]: starting. 2024-09-13T11:53:43.841 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34802]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-34765 2024-09-13T11:53:43.841 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34802]: created object 0... 2024-09-13T11:53:43.841 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34802]: created object 25... 2024-09-13T11:53:43.841 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34802]: created object 49... 2024-09-13T11:53:43.842 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34802]: finishing. 2024-09-13T11:53:43.842 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34802]: shutting down. 2024-09-13T11:53:43.855 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:43.856 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[34804]: starting. 2024-09-13T11:53:43.856 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[34804]: listing objects. 2024-09-13T11:53:43.856 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[34804]: listed object 0... 2024-09-13T11:53:43.856 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[34804]: listed object 25... 2024-09-13T11:53:43.856 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[34804]: saw 50 objects 2024-09-13T11:53:43.856 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[34804]: shutting down. 2024-09-13T11:53:44.441 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:44.442 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[34803]: starting. 2024-09-13T11:53:44.442 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[34803]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-34765 2024-09-13T11:53:44.442 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[34803]: shutting down. 2024-09-13T11:53:44.443 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:44.443 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:44.443 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:53:44.446 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:53:46.564 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34873]: starting. 2024-09-13T11:53:46.564 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34873]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-34872 2024-09-13T11:53:46.564 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34873]: created object 0... 2024-09-13T11:53:46.564 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34873]: created object 25... 2024-09-13T11:53:46.564 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34873]: created object 49... 2024-09-13T11:53:46.564 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34873]: finishing. 2024-09-13T11:53:46.564 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34873]: shutting down. 2024-09-13T11:53:47.469 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[34874]: starting. 2024-09-13T11:53:47.469 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[34874]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-34872 2024-09-13T11:53:47.469 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[34874]: shutting down. 2024-09-13T11:53:49.569 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:49.569 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34909]: starting. 2024-09-13T11:53:49.569 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34909]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-34872 2024-09-13T11:53:49.569 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34909]: created object 0... 2024-09-13T11:53:49.569 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34909]: created object 25... 2024-09-13T11:53:49.569 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34909]: created object 49... 2024-09-13T11:53:49.569 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34909]: finishing. 2024-09-13T11:53:49.569 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[34909]: shutting down. 2024-09-13T11:53:49.581 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:49.581 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[34911]: starting. 2024-09-13T11:53:49.581 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[34911]: listing objects. 2024-09-13T11:53:49.582 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[34911]: listed object 0... 2024-09-13T11:53:49.582 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[34911]: listed object 25... 2024-09-13T11:53:49.582 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[34911]: saw 50 objects 2024-09-13T11:53:49.582 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[34911]: shutting down. 2024-09-13T11:53:50.477 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:50.477 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[34910]: starting. 2024-09-13T11:53:50.477 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[34910]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-34872 2024-09-13T11:53:50.478 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[34910]: shutting down. 2024-09-13T11:53:50.479 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:50.479 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:50.479 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:53:50.482 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:53:52.076 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34980]: starting. 2024-09-13T11:53:52.076 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34980]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-34979 2024-09-13T11:53:52.076 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34980]: created object 0... 2024-09-13T11:53:52.076 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34980]: created object 25... 2024-09-13T11:53:52.076 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34980]: created object 49... 2024-09-13T11:53:52.076 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34980]: finishing. 2024-09-13T11:53:52.076 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[34980]: shutting down. 2024-09-13T11:53:53.000 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[34981]: starting. 2024-09-13T11:53:53.001 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[34981]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-34979 2024-09-13T11:53:53.001 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[34981]: shutting down. 2024-09-13T11:53:55.099 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:55.099 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35016]: starting. 2024-09-13T11:53:55.099 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35016]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-34979 2024-09-13T11:53:55.099 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35016]: created object 0... 2024-09-13T11:53:55.099 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35016]: created object 25... 2024-09-13T11:53:55.100 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35016]: created object 49... 2024-09-13T11:53:55.100 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35016]: finishing. 2024-09-13T11:53:55.100 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35016]: shutting down. 2024-09-13T11:53:55.111 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:55.111 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35018]: starting. 2024-09-13T11:53:55.111 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35018]: listing objects. 2024-09-13T11:53:55.111 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35018]: listed object 0... 2024-09-13T11:53:55.111 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35018]: listed object 25... 2024-09-13T11:53:55.111 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35018]: saw 50 objects 2024-09-13T11:53:55.111 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35018]: shutting down. 2024-09-13T11:53:55.988 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:55.988 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35017]: starting. 2024-09-13T11:53:55.988 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35017]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-34979 2024-09-13T11:53:55.988 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35017]: shutting down. 2024-09-13T11:53:55.989 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:55.990 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:53:55.990 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:53:55.992 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:53:58.111 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35087]: starting. 2024-09-13T11:53:58.111 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35087]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-35086 2024-09-13T11:53:58.111 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35087]: created object 0... 2024-09-13T11:53:58.111 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35087]: created object 25... 2024-09-13T11:53:58.111 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35087]: created object 49... 2024-09-13T11:53:58.111 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35087]: finishing. 2024-09-13T11:53:58.111 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35087]: shutting down. 2024-09-13T11:53:59.026 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35088]: starting. 2024-09-13T11:53:59.026 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35088]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-35086 2024-09-13T11:53:59.026 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35088]: shutting down. 2024-09-13T11:54:01.137 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:01.137 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35123]: starting. 2024-09-13T11:54:01.137 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35123]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-35086 2024-09-13T11:54:01.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35123]: created object 0... 2024-09-13T11:54:01.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35123]: created object 25... 2024-09-13T11:54:01.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35123]: created object 49... 2024-09-13T11:54:01.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35123]: finishing. 2024-09-13T11:54:01.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35123]: shutting down. 2024-09-13T11:54:01.149 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:01.150 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35125]: starting. 2024-09-13T11:54:01.150 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35125]: listing objects. 2024-09-13T11:54:01.150 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35125]: listed object 0... 2024-09-13T11:54:01.150 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35125]: listed object 25... 2024-09-13T11:54:01.150 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35125]: saw 50 objects 2024-09-13T11:54:01.150 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35125]: shutting down. 2024-09-13T11:54:02.049 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:02.050 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35124]: starting. 2024-09-13T11:54:02.050 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35124]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-35086 2024-09-13T11:54:02.050 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35124]: shutting down. 2024-09-13T11:54:02.051 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:02.051 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:02.051 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:54:02.053 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:54:04.148 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35194]: starting. 2024-09-13T11:54:04.148 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35194]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-35193 2024-09-13T11:54:04.148 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35194]: created object 0... 2024-09-13T11:54:04.149 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35194]: created object 25... 2024-09-13T11:54:04.149 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35194]: created object 49... 2024-09-13T11:54:04.149 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35194]: finishing. 2024-09-13T11:54:04.149 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35194]: shutting down. 2024-09-13T11:54:05.059 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35195]: starting. 2024-09-13T11:54:05.059 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35195]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-35193 2024-09-13T11:54:05.059 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35195]: shutting down. 2024-09-13T11:54:07.090 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:07.090 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35230]: starting. 2024-09-13T11:54:07.090 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35230]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-35193 2024-09-13T11:54:07.090 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35230]: created object 0... 2024-09-13T11:54:07.090 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35230]: created object 25... 2024-09-13T11:54:07.090 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35230]: created object 49... 2024-09-13T11:54:07.090 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35230]: finishing. 2024-09-13T11:54:07.091 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35230]: shutting down. 2024-09-13T11:54:07.102 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:07.102 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35232]: starting. 2024-09-13T11:54:07.102 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35232]: listing objects. 2024-09-13T11:54:07.102 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35232]: listed object 0... 2024-09-13T11:54:07.102 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35232]: listed object 25... 2024-09-13T11:54:07.102 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35232]: saw 50 objects 2024-09-13T11:54:07.102 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35232]: shutting down. 2024-09-13T11:54:07.994 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:07.994 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35231]: starting. 2024-09-13T11:54:07.994 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35231]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-35193 2024-09-13T11:54:07.994 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35231]: shutting down. 2024-09-13T11:54:07.996 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:07.996 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:07.996 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:54:07.999 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:54:10.089 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35301]: starting. 2024-09-13T11:54:10.089 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35301]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-35300 2024-09-13T11:54:10.089 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35301]: created object 0... 2024-09-13T11:54:10.089 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35301]: created object 25... 2024-09-13T11:54:10.089 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35301]: created object 49... 2024-09-13T11:54:10.090 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35301]: finishing. 2024-09-13T11:54:10.090 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35301]: shutting down. 2024-09-13T11:54:11.000 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35302]: starting. 2024-09-13T11:54:11.000 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35302]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-35300 2024-09-13T11:54:11.000 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35302]: shutting down. 2024-09-13T11:54:13.307 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:13.307 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35337]: starting. 2024-09-13T11:54:13.307 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35337]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-35300 2024-09-13T11:54:13.307 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35337]: created object 0... 2024-09-13T11:54:13.307 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35337]: created object 25... 2024-09-13T11:54:13.307 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35337]: created object 49... 2024-09-13T11:54:13.307 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35337]: finishing. 2024-09-13T11:54:13.307 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35337]: shutting down. 2024-09-13T11:54:13.321 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:13.321 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35339]: starting. 2024-09-13T11:54:13.321 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35339]: listing objects. 2024-09-13T11:54:13.321 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35339]: listed object 0... 2024-09-13T11:54:13.321 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35339]: listed object 25... 2024-09-13T11:54:13.322 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35339]: saw 50 objects 2024-09-13T11:54:13.322 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35339]: shutting down. 2024-09-13T11:54:14.020 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:14.020 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35338]: starting. 2024-09-13T11:54:14.020 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35338]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-35300 2024-09-13T11:54:14.021 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35338]: shutting down. 2024-09-13T11:54:14.025 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:14.025 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:14.026 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:54:14.026 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:54:16.086 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35408]: starting. 2024-09-13T11:54:16.086 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35408]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-35407 2024-09-13T11:54:16.086 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35408]: created object 0... 2024-09-13T11:54:16.086 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35408]: created object 25... 2024-09-13T11:54:16.086 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35408]: created object 49... 2024-09-13T11:54:16.086 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35408]: finishing. 2024-09-13T11:54:16.086 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35408]: shutting down. 2024-09-13T11:54:17.000 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35409]: starting. 2024-09-13T11:54:17.000 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35409]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-35407 2024-09-13T11:54:17.001 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35409]: shutting down. 2024-09-13T11:54:19.086 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:19.086 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35444]: starting. 2024-09-13T11:54:19.086 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35444]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-35407 2024-09-13T11:54:19.086 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35444]: created object 0... 2024-09-13T11:54:19.086 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35444]: created object 25... 2024-09-13T11:54:19.086 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35444]: created object 49... 2024-09-13T11:54:19.086 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35444]: finishing. 2024-09-13T11:54:19.086 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35444]: shutting down. 2024-09-13T11:54:19.098 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:19.098 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35446]: starting. 2024-09-13T11:54:19.098 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35446]: listing objects. 2024-09-13T11:54:19.098 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35446]: listed object 0... 2024-09-13T11:54:19.098 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35446]: listed object 25... 2024-09-13T11:54:19.098 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35446]: saw 50 objects 2024-09-13T11:54:19.098 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35446]: shutting down. 2024-09-13T11:54:20.022 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:20.022 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35445]: starting. 2024-09-13T11:54:20.022 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35445]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-35407 2024-09-13T11:54:20.023 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35445]: shutting down. 2024-09-13T11:54:20.024 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:20.024 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:20.024 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:54:20.027 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:54:22.103 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35515]: starting. 2024-09-13T11:54:22.104 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35515]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-35514 2024-09-13T11:54:22.104 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35515]: created object 0... 2024-09-13T11:54:22.104 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35515]: created object 25... 2024-09-13T11:54:22.104 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35515]: created object 49... 2024-09-13T11:54:22.104 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35515]: finishing. 2024-09-13T11:54:22.104 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35515]: shutting down. 2024-09-13T11:54:23.029 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35516]: starting. 2024-09-13T11:54:23.029 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35516]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-35514 2024-09-13T11:54:23.029 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35516]: shutting down. 2024-09-13T11:54:25.127 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:25.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35551]: starting. 2024-09-13T11:54:25.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35551]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-35514 2024-09-13T11:54:25.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35551]: created object 0... 2024-09-13T11:54:25.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35551]: created object 25... 2024-09-13T11:54:25.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35551]: created object 49... 2024-09-13T11:54:25.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35551]: finishing. 2024-09-13T11:54:25.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35551]: shutting down. 2024-09-13T11:54:25.140 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:25.140 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35553]: starting. 2024-09-13T11:54:25.140 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35553]: listing objects. 2024-09-13T11:54:25.140 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35553]: listed object 0... 2024-09-13T11:54:25.140 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35553]: listed object 25... 2024-09-13T11:54:25.140 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35553]: saw 50 objects 2024-09-13T11:54:25.141 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35553]: shutting down. 2024-09-13T11:54:25.995 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:25.995 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35552]: starting. 2024-09-13T11:54:25.995 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35552]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-35514 2024-09-13T11:54:25.996 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35552]: shutting down. 2024-09-13T11:54:25.996 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:25.997 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:25.997 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:54:25.999 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:54:28.154 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35622]: starting. 2024-09-13T11:54:28.154 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35622]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-35621 2024-09-13T11:54:28.155 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35622]: created object 0... 2024-09-13T11:54:28.155 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35622]: created object 25... 2024-09-13T11:54:28.155 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35622]: created object 49... 2024-09-13T11:54:28.155 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35622]: finishing. 2024-09-13T11:54:28.155 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35622]: shutting down. 2024-09-13T11:54:29.075 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35623]: starting. 2024-09-13T11:54:29.075 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35623]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-35621 2024-09-13T11:54:29.075 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35623]: shutting down. 2024-09-13T11:54:31.172 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:31.172 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35658]: starting. 2024-09-13T11:54:31.172 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35658]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-35621 2024-09-13T11:54:31.172 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35658]: created object 0... 2024-09-13T11:54:31.173 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35658]: created object 25... 2024-09-13T11:54:31.173 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35658]: created object 49... 2024-09-13T11:54:31.173 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35658]: finishing. 2024-09-13T11:54:31.173 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35658]: shutting down. 2024-09-13T11:54:31.185 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:31.186 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35660]: starting. 2024-09-13T11:54:31.186 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35660]: listing objects. 2024-09-13T11:54:31.186 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35660]: listed object 0... 2024-09-13T11:54:31.186 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35660]: listed object 25... 2024-09-13T11:54:31.186 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35660]: saw 50 objects 2024-09-13T11:54:31.186 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35660]: shutting down. 2024-09-13T11:54:32.092 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:32.092 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35659]: starting. 2024-09-13T11:54:32.092 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35659]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-35621 2024-09-13T11:54:32.092 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35659]: shutting down. 2024-09-13T11:54:32.094 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:32.094 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:32.094 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:54:32.096 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:54:34.197 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35729]: starting. 2024-09-13T11:54:34.197 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35729]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-35728 2024-09-13T11:54:34.197 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35729]: created object 0... 2024-09-13T11:54:34.197 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35729]: created object 25... 2024-09-13T11:54:34.198 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35729]: created object 49... 2024-09-13T11:54:34.198 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35729]: finishing. 2024-09-13T11:54:34.198 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35729]: shutting down. 2024-09-13T11:54:35.111 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35730]: starting. 2024-09-13T11:54:35.111 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35730]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-35728 2024-09-13T11:54:35.111 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35730]: shutting down. 2024-09-13T11:54:37.088 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:37.088 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35765]: starting. 2024-09-13T11:54:37.088 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35765]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-35728 2024-09-13T11:54:37.088 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35765]: created object 0... 2024-09-13T11:54:37.088 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35765]: created object 25... 2024-09-13T11:54:37.088 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35765]: created object 49... 2024-09-13T11:54:37.088 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35765]: finishing. 2024-09-13T11:54:37.088 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35765]: shutting down. 2024-09-13T11:54:37.097 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:37.097 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35767]: starting. 2024-09-13T11:54:37.098 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35767]: listing objects. 2024-09-13T11:54:37.098 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35767]: listed object 0... 2024-09-13T11:54:37.098 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35767]: listed object 25... 2024-09-13T11:54:37.098 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35767]: saw 50 objects 2024-09-13T11:54:37.098 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35767]: shutting down. 2024-09-13T11:54:38.000 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:38.001 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35766]: starting. 2024-09-13T11:54:38.001 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35766]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-35728 2024-09-13T11:54:38.001 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35766]: shutting down. 2024-09-13T11:54:38.002 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:38.002 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:38.002 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:54:38.004 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:54:40.081 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35836]: starting. 2024-09-13T11:54:40.081 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35836]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-35835 2024-09-13T11:54:40.081 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35836]: created object 0... 2024-09-13T11:54:40.081 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35836]: created object 25... 2024-09-13T11:54:40.081 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35836]: created object 49... 2024-09-13T11:54:40.081 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35836]: finishing. 2024-09-13T11:54:40.081 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35836]: shutting down. 2024-09-13T11:54:41.008 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35837]: starting. 2024-09-13T11:54:41.009 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35837]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-35835 2024-09-13T11:54:41.009 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35837]: shutting down. 2024-09-13T11:54:43.093 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:43.093 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35872]: starting. 2024-09-13T11:54:43.093 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35872]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-35835 2024-09-13T11:54:43.093 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35872]: created object 0... 2024-09-13T11:54:43.094 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35872]: created object 25... 2024-09-13T11:54:43.094 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35872]: created object 49... 2024-09-13T11:54:43.094 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35872]: finishing. 2024-09-13T11:54:43.094 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35872]: shutting down. 2024-09-13T11:54:43.105 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:43.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35874]: starting. 2024-09-13T11:54:43.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35874]: listing objects. 2024-09-13T11:54:43.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35874]: listed object 0... 2024-09-13T11:54:43.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35874]: listed object 25... 2024-09-13T11:54:43.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35874]: saw 50 objects 2024-09-13T11:54:43.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35874]: shutting down. 2024-09-13T11:54:44.017 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:44.017 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35873]: starting. 2024-09-13T11:54:44.017 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35873]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-35835 2024-09-13T11:54:44.017 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35873]: shutting down. 2024-09-13T11:54:44.018 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:44.018 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:44.018 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:54:44.021 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:54:46.077 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35943]: starting. 2024-09-13T11:54:46.077 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35943]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-35942 2024-09-13T11:54:46.077 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35943]: created object 0... 2024-09-13T11:54:46.077 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35943]: created object 25... 2024-09-13T11:54:46.078 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35943]: created object 49... 2024-09-13T11:54:46.078 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35943]: finishing. 2024-09-13T11:54:46.078 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[35943]: shutting down. 2024-09-13T11:54:47.001 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35944]: starting. 2024-09-13T11:54:47.001 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35944]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-35942 2024-09-13T11:54:47.001 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[35944]: shutting down. 2024-09-13T11:54:49.098 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:49.098 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35979]: starting. 2024-09-13T11:54:49.098 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35979]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-35942 2024-09-13T11:54:49.099 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35979]: created object 0... 2024-09-13T11:54:49.099 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35979]: created object 25... 2024-09-13T11:54:49.099 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35979]: created object 49... 2024-09-13T11:54:49.099 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35979]: finishing. 2024-09-13T11:54:49.099 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[35979]: shutting down. 2024-09-13T11:54:49.110 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:49.111 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35981]: starting. 2024-09-13T11:54:49.111 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35981]: listing objects. 2024-09-13T11:54:49.111 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35981]: listed object 0... 2024-09-13T11:54:49.111 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35981]: listed object 25... 2024-09-13T11:54:49.111 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35981]: saw 50 objects 2024-09-13T11:54:49.111 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[35981]: shutting down. 2024-09-13T11:54:50.023 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:50.023 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35980]: starting. 2024-09-13T11:54:50.023 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35980]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-35942 2024-09-13T11:54:50.023 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[35980]: shutting down. 2024-09-13T11:54:50.024 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:50.024 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:50.024 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:54:50.027 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:54:52.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36050]: starting. 2024-09-13T11:54:52.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36050]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-36049 2024-09-13T11:54:52.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36050]: created object 0... 2024-09-13T11:54:52.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36050]: created object 25... 2024-09-13T11:54:52.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36050]: created object 49... 2024-09-13T11:54:52.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36050]: finishing. 2024-09-13T11:54:52.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36050]: shutting down. 2024-09-13T11:54:53.032 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36051]: starting. 2024-09-13T11:54:53.032 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36051]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-36049 2024-09-13T11:54:53.032 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36051]: shutting down. 2024-09-13T11:54:55.129 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:55.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36086]: starting. 2024-09-13T11:54:55.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36086]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-36049 2024-09-13T11:54:55.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36086]: created object 0... 2024-09-13T11:54:55.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36086]: created object 25... 2024-09-13T11:54:55.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36086]: created object 49... 2024-09-13T11:54:55.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36086]: finishing. 2024-09-13T11:54:55.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36086]: shutting down. 2024-09-13T11:54:55.141 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:55.142 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36088]: starting. 2024-09-13T11:54:55.142 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36088]: listing objects. 2024-09-13T11:54:55.142 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36088]: listed object 0... 2024-09-13T11:54:55.142 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36088]: listed object 25... 2024-09-13T11:54:55.142 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36088]: saw 50 objects 2024-09-13T11:54:55.142 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36088]: shutting down. 2024-09-13T11:54:56.000 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:56.000 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36087]: starting. 2024-09-13T11:54:56.001 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36087]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-36049 2024-09-13T11:54:56.001 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36087]: shutting down. 2024-09-13T11:54:56.002 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:56.002 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:54:56.002 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:54:56.004 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:54:58.131 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36157]: starting. 2024-09-13T11:54:58.131 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36157]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-36156 2024-09-13T11:54:58.131 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36157]: created object 0... 2024-09-13T11:54:58.131 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36157]: created object 25... 2024-09-13T11:54:58.132 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36157]: created object 49... 2024-09-13T11:54:58.132 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36157]: finishing. 2024-09-13T11:54:58.132 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36157]: shutting down. 2024-09-13T11:54:58.903 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-13T11:54:58.903 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-13T11:54:58.904 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-13T11:54:58.906 INFO:tasks.ceph.mon.a:Started 2024-09-13T11:54:58.907 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-13T11:54:58.907 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:54:59.003 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:54:59.002+0000 7fb512b32c80 -1 mon.a@-1(???).log v0 log_external_backlog rewinding external_log_to from 26 -> 0 (sync_force? mon rebuild?) 2024-09-13T11:54:59.015 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.014+0000 7f0fe7dd1640 10 monclient: get_monmap_and_config 2024-09-13T11:54:59.015 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.014+0000 7f0fe7dd1640 10 monclient: build_initial_monmap 2024-09-13T11:54:59.015 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.014+0000 7f0fe7dd1640 10 monclient: monmap: 2024-09-13T11:54:59.015 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:54:59.015 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.015 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:54:59.015037+0000 2024-09-13T11:54:59.015 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:54:59.015037+0000 2024-09-13T11:54:59.016 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:54:59.016 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.016 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:54:59.016 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:54:59.016 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:54:59.016 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:54:59.016 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:54:59.016 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.014+0000 7f0fe7dd1640 1 Processor -- start 2024-09-13T11:54:59.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe7dd1640 1 -- start start 2024-09-13T11:54:59.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe7dd1640 10 monclient: init 2024-09-13T11:54:59.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe7dd1640 10 monclient: _reopen_session rank -1 2024-09-13T11:54:59.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe7dd1640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-13T11:54:59.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe7dd1640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0fe0137a60 0x7f0fe00a2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe7dd1640 10 monclient(hunting): picked mon.noname-a con 0x7f0fe0137a60 addr v2:172.21.15.100:3300/0 2024-09-13T11:54:59.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe7dd1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0fe0137e40 addr v1:172.21.15.100:6789/0 2024-09-13T11:54:59.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe7dd1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0fe0138250 addr v1:172.21.15.122:6789/0 2024-09-13T11:54:59.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe7dd1640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00a5770 con 0x7f0fe0137e40 2024-09-13T11:54:59.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe7dd1640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00a5be0 con 0x7f0fe0138250 2024-09-13T11:54:59.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe7dd1640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe7dd1640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00a4410 con 0x7f0fe0137a60 2024-09-13T11:54:59.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe7dd1640 10 monclient(hunting): _renew_subs 2024-09-13T11:54:59.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe7dd1640 10 monclient(hunting): authenticate will time out at 879.608595s 2024-09-13T11:54:59.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe5b6d640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0fe0137a60 0x7f0fe00a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe536c640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f0fe0137e40 0x7f0fe00a4950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:48206/0 (socket says 172.21.15.100:48206) 2024-09-13T11:54:59.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe5b6d640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0fe0137a60 0x7f0fe00a2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53348/0 (socket says 172.21.15.100:53348) 2024-09-13T11:54:59.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe536c640 1 -- 172.21.15.100:0/2953287760 learned_addr learned my addr 172.21.15.100:0/2953287760 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:54:59.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe5b6d640 10 monclient(hunting): get_auth_request con 0x7f0fe0137a60 auth_method 0 2024-09-13T11:54:59.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe5b6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe5b6d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe5b6d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe5b6d640 1 -- 172.21.15.100:0/2953287760 >> v2:172.21.15.100:3300/0 conn(0x7f0fe0137a60 msgr2=0x7f0fe00a2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-13T11:54:59.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe5b6d640 1 -- 172.21.15.100:0/2953287760 >> v2:172.21.15.100:3300/0 conn(0x7f0fe0137a60 msgr2=0x7f0fe00a2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:54:59.029 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe5b6d640 1 --2- 172.21.15.100:0/2953287760 >> v2:172.21.15.100:3300/0 conn(0x7f0fe0137a60 0x7f0fe00a2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:54:59.029 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.015+0000 7f0fe5b6d640 1 --2- 172.21.15.100:0/2953287760 >> v2:172.21.15.100:3300/0 conn(0x7f0fe0137a60 0x7f0fe00a2980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:54:59.029 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 1 -- 172.21.15.100:0/2953287760 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4156922010 0 0) 0x7f0fe00a5be0 con 0x7f0fe0138250 2024-09-13T11:54:59.029 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 10 monclient(hunting): _negotiate 2024-09-13T11:54:59.029 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.029 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.029 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 10 monclient(hunting): my global_id is 4918 2024-09-13T11:54:59.029 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 1 -- 172.21.15.100:0/2953287760 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fcc003520 con 0x7f0fe0138250 2024-09-13T11:54:59.029 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 1 -- 172.21.15.100:0/2953287760 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3919355466 0 0) 0x7f0fcc003520 con 0x7f0fe0138250 2024-09-13T11:54:59.029 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 1 -- 172.21.15.100:0/2953287760 >> v2:172.21.15.100:3300/0 conn(0x7f0fe0137a60 msgr2=0x7f0fe00a2980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:54:59.029 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 1 --2- 172.21.15.100:0/2953287760 >> v2:172.21.15.100:3300/0 conn(0x7f0fe0137a60 0x7f0fe00a2980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.029 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 1 -- 172.21.15.100:0/2953287760 >> v1:172.21.15.100:6789/0 conn(0x7f0fe0137e40 legacy=0x7f0fe00a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.029 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 10 monclient: _finish_hunting 0 2024-09-13T11:54:59.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 1 monclient: found mon.noname-c-legacy 2024-09-13T11:54:59.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:54:59.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.122:6789/0 2024-09-13T11:54:59.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 1 -- 172.21.15.100:0/2953287760 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fe00a76b0 con 0x7f0fe0138250 2024-09-13T11:54:59.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 10 monclient: _finish_auth 0 2024-09-13T11:54:59.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 10 monclient: _check_auth_tickets 2024-09-13T11:54:59.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:54:59.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe7dd1640 5 monclient: authenticate success, global_id 4918 2024-09-13T11:54:59.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe7dd1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:54:59.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 1 -- 172.21.15.100:0/2953287760 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 626700363 0 0) 0x7f0fd8001a70 con 0x7f0fe0138250 2024-09-13T11:54:59.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:54:59.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-13T11:54:59.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 10 monclient: dump: 2024-09-13T11:54:59.031 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:54:59.031 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.031 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.031 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.031 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:54:59.031 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.031 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:54:59.031 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:54:59.031 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:54:59.031 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.031 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] but i'm connected to v1:172.21.15.122:6789/0, reconnecting 2024-09-13T11:54:59.031 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 10 monclient: _reopen_session rank -1 2024-09-13T11:54:59.031 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 1 -- 172.21.15.100:0/2953287760 >> v1:172.21.15.122:6789/0 conn(0x7f0fe0138250 legacy=0x7f0fe00a5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:54:59.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 1 --2- 172.21.15.100:0/2953287760 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fcc003a50 0x7f0fcc003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 10 monclient(hunting): picked mon.c con 0x7f0fcc003a50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:54:59.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 1 --2- 172.21.15.100:0/2953287760 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fcc004800 0x7f0fcc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 10 monclient(hunting): picked mon.a con 0x7f0fcc004800 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:54:59.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 1 --2- 172.21.15.100:0/2953287760 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0fcc0053c0 0x7f0fcc00ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 10 monclient(hunting): picked mon.b con 0x7f0fcc0053c0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:54:59.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 1 -- 172.21.15.100:0/2953287760 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f0fcc00c2d0 con 0x7f0fcc004800 2024-09-13T11:54:59.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 1 -- 172.21.15.100:0/2953287760 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f0fcc00c480 con 0x7f0fcc0053c0 2024-09-13T11:54:59.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.033 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 1 -- 172.21.15.100:0/2953287760 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f0fcc00c680 con 0x7f0fcc003a50 2024-09-13T11:54:59.033 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 10 monclient(hunting): _renew_subs 2024-09-13T11:54:59.033 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe4b6b640 10 monclient(hunting): _finish_auth 0 2024-09-13T11:54:59.033 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe7dd1640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-13T11:54:59.033 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe5b6d640 1 --2- 172.21.15.100:0/2953287760 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fcc003a50 0x7f0fcc003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.033 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe536c640 1 --2- 172.21.15.100:0/2953287760 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fcc004800 0x7f0fcc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.033 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe5b6d640 10 monclient(hunting): get_auth_request con 0x7f0fcc003a50 auth_method 0 2024-09-13T11:54:59.033 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe5b6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.033 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe5b6d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.033 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe5b6d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:54:59.033 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe536c640 10 monclient(hunting): get_auth_request con 0x7f0fcc004800 auth_method 0 2024-09-13T11:54:59.033 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe536c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.033 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe536c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.033 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe536c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:54:59.034 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe5b6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:54:59.034 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe5b6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:54:59.034 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe636e640 1 --2- 172.21.15.100:0/2953287760 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0fcc0053c0 0x7f0fcc00ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.034 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe5b6d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:54:59.034 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe636e640 10 monclient(hunting): get_auth_request con 0x7f0fcc0053c0 auth_method 0 2024-09-13T11:54:59.034 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe636e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.034 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe636e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.034 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe636e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:54:59.034 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe536c640 1 -- 172.21.15.100:0/2953287760 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fcc004800 msgr2=0x7f0fcc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T11:54:59.034 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe536c640 1 -- 172.21.15.100:0/2953287760 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fcc004800 msgr2=0x7f0fcc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:54:59.034 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe536c640 1 --2- 172.21.15.100:0/2953287760 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fcc004800 0x7f0fcc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:54:59.034 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe536c640 1 --2- 172.21.15.100:0/2953287760 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fcc004800 0x7f0fcc004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:54:59.034 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe636e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:54:59.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe636e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:54:59.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe636e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:54:59.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe5b6d640 10 monclient(hunting): handle_auth_done global_id 4918 payload 293 2024-09-13T11:54:59.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe5b6d640 1 -- 172.21.15.100:0/2953287760 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0fcc0053c0 msgr2=0x7f0fcc00ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:54:59.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe5b6d640 1 --2- 172.21.15.100:0/2953287760 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0fcc0053c0 0x7f0fcc00ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe5b6d640 1 -- 172.21.15.100:0/2953287760 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fcc004800 msgr2=0x7f0fcc004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:54:59.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe5b6d640 1 --2- 172.21.15.100:0/2953287760 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fcc004800 0x7f0fcc004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe5b6d640 10 monclient: _finish_hunting 0 2024-09-13T11:54:59.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe5b6d640 1 monclient: found mon.c 2024-09-13T11:54:59.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe5b6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:54:59.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe5b6d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:54:59.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe5b6d640 1 -- 172.21.15.100:0/2953287760 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fcc003520 con 0x7f0fcc003a50 2024-09-13T11:54:59.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe5b6d640 10 monclient: _finish_auth 0 2024-09-13T11:54:59.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe5b6d640 10 monclient: _check_auth_tickets 2024-09-13T11:54:59.036 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe5b6d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:54:59.036 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.016+0000 7f0fe636e640 1 -- 172.21.15.100:0/2953287760 reap_dead start 2024-09-13T11:54:59.036 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.017+0000 7f0fe5b6d640 1 --2- 172.21.15.100:0/2953287760 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fcc003a50 0x7f0fcc003e90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc00b9e0 tx=0x7f0fcc005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93ef4aeb36848ddd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:54:59.036 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.017+0000 7f0fe4b6b640 1 -- 172.21.15.100:0/2953287760 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc030070 con 0x7f0fcc003a50 2024-09-13T11:54:59.036 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.017+0000 7f0fe4b6b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:54:59.036 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.017+0000 7f0fe4b6b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:54:59.036 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.017+0000 7f0fe4b6b640 10 monclient: dump: 2024-09-13T11:54:59.036 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:54:59.036 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.036 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.036 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.036 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:54:59.036 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.037 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:54:59.037 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:54:59.037 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:54:59.037 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.037 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.017+0000 7f0fe4b6b640 1 -- 172.21.15.100:0/2953287760 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fdc02ada0 con 0x7f0fcc003a50 2024-09-13T11:54:59.037 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.017+0000 7f0fe7dd1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:54:59.037 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.017+0000 7f0fe4b6b640 10 monclient: handle_config config(0 keys) 2024-09-13T11:54:59.037 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.017+0000 7f0fe4b6b640 1 -- 172.21.15.100:0/2953287760 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc00b5e0 con 0x7f0fcc003a50 2024-09-13T11:54:59.037 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.017+0000 7f0fe7dd1640 10 monclient: get_monmap_and_config success 2024-09-13T11:54:59.037 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.017+0000 7f0fe7dd1640 4 set_mon_vals no callback set 2024-09-13T11:54:59.037 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.017+0000 7f0fe4b6b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:54:59.037 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.017+0000 7f0fe4b6b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:54:59.037 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.017+0000 7f0fe4b6b640 10 monclient: dump: 2024-09-13T11:54:59.037 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:54:59.038 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.038 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.038 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.038 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:54:59.038 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.038 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:54:59.038 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:54:59.038 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:54:59.038 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.038 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.018+0000 7f0fe7dd1640 10 monclient: shutdown 2024-09-13T11:54:59.038 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.018+0000 7f0fe7dd1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:54:59.038 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.018+0000 7f0fe7dd1640 1 -- 172.21.15.100:0/2953287760 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fcc003a50 msgr2=0x7f0fcc003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.038 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.018+0000 7f0fe7dd1640 1 --2- 172.21.15.100:0/2953287760 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fcc003a50 0x7f0fcc003e90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc00b9e0 tx=0x7f0fcc005830 comp rx=0 tx=0).stop 2024-09-13T11:54:59.039 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.018+0000 7f0fe7dd1640 1 -- 172.21.15.100:0/2953287760 shutdown_connections 2024-09-13T11:54:59.039 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.018+0000 7f0fe7dd1640 1 --2- 172.21.15.100:0/2953287760 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fcc003a50 0x7f0fcc003e90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.039 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.018+0000 7f0fe7dd1640 1 -- 172.21.15.100:0/2953287760 >> 172.21.15.100:0/2953287760 conn(0x7f0fe00aa580 msgr2=0x7f0fe00a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:54:59.039 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.018+0000 7f0fe7dd1640 1 -- 172.21.15.100:0/2953287760 shutdown_connections 2024-09-13T11:54:59.039 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.018+0000 7f0fe7dd1640 1 -- 172.21.15.100:0/2953287760 wait complete. 2024-09-13T11:54:59.039 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.019+0000 7f0fe7dd1640 10 monclient: build_initial_monmap 2024-09-13T11:54:59.039 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.019+0000 7f0fe7dd1640 10 monclient: monmap: 2024-09-13T11:54:59.039 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:54:59.039 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:54:59.039 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:54:59.039 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:54:59.039 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:54:59.039 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.040 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:54:59.040 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:54:59.040 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:54:59.040 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.019+0000 7f0fe7dd1640 1 Processor -- start 2024-09-13T11:54:59.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.019+0000 7f0fe7dd1640 1 -- start start 2024-09-13T11:54:59.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.019+0000 7f0fe7dd1640 10 monclient: init 2024-09-13T11:54:59.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.019+0000 7f0fe7dd1640 10 monclient: _reopen_session rank -1 2024-09-13T11:54:59.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.019+0000 7f0fe7dd1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:54:59.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.019+0000 7f0fe7dd1640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fcc003a50 0x7f0fe01cc770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.019+0000 7f0fe7dd1640 10 monclient(hunting): picked mon.noname-c con 0x7f0fcc003a50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:54:59.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.019+0000 7f0fe7dd1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0fe0126860 0x7f0fe01cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.019+0000 7f0fe7dd1640 10 monclient(hunting): picked mon.noname-b con 0x7f0fe0126860 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:54:59.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.019+0000 7f0fe7dd1640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fe0124290 0x7f0fe0124740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.019+0000 7f0fe7dd1640 10 monclient(hunting): picked mon.noname-a con 0x7f0fe0124290 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:54:59.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.019+0000 7f0fe7dd1640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.019+0000 7f0fe7dd1640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe006fd80 con 0x7f0fe0124290 2024-09-13T11:54:59.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.019+0000 7f0fe7dd1640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.019+0000 7f0fe7dd1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe00a1700 con 0x7f0fe0126860 2024-09-13T11:54:59.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.019+0000 7f0fe7dd1640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.019+0000 7f0fe7dd1640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f0fe006ff00 con 0x7f0fcc003a50 2024-09-13T11:54:59.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe7dd1640 10 monclient(hunting): _renew_subs 2024-09-13T11:54:59.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe7dd1640 10 monclient(hunting): authenticate will time out at 879.613014s 2024-09-13T11:54:59.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe5b6d640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fcc003a50 0x7f0fe01cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe636e640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fe0124290 0x7f0fe0124740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe5b6d640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fcc003a50 0x7f0fe01cc770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:36822/0 (socket says 172.21.15.100:36822) 2024-09-13T11:54:59.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe5b6d640 1 -- 172.21.15.100:0/301100211 learned_addr learned my addr 172.21.15.100:0/301100211 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:54:59.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe5b6d640 10 monclient(hunting): get_auth_request con 0x7f0fcc003a50 auth_method 0 2024-09-13T11:54:59.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe5b6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe5b6d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe5b6d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe636e640 10 monclient(hunting): get_auth_request con 0x7f0fe0124290 auth_method 0 2024-09-13T11:54:59.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe636e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe636e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe636e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe5b6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:54:59.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe5b6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:54:59.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe5b6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:54:59.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe636e640 1 -- 172.21.15.100:0/301100211 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fe0124290 msgr2=0x7f0fe0124740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-13T11:54:59.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe636e640 1 -- 172.21.15.100:0/301100211 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fe0124290 msgr2=0x7f0fe0124740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T11:54:59.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe636e640 1 --2- 172.21.15.100:0/301100211 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fe0124290 0x7f0fe0124740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:54:59.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe636e640 1 --2- 172.21.15.100:0/301100211 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fe0124290 0x7f0fe0124740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:54:59.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe536c640 1 --2- 172.21.15.100:0/301100211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0fe0126860 0x7f0fe01cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe5b6d640 10 monclient(hunting): handle_auth_done global_id 4814 payload 824 2024-09-13T11:54:59.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe5b6d640 1 -- 172.21.15.100:0/301100211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0fe0126860 msgr2=0x7f0fe01cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe5b6d640 1 --2- 172.21.15.100:0/301100211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0fe0126860 0x7f0fe01cccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe5b6d640 1 -- 172.21.15.100:0/301100211 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fe0124290 msgr2=0x7f0fe0124740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:54:59.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe5b6d640 1 --2- 172.21.15.100:0/301100211 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fe0124290 0x7f0fe0124740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe5b6d640 10 monclient: _finish_hunting 0 2024-09-13T11:54:59.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe5b6d640 1 monclient: found mon.noname-c 2024-09-13T11:54:59.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe5b6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:54:59.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe5b6d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:54:59.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe5b6d640 1 -- 172.21.15.100:0/301100211 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fe00a0770 con 0x7f0fcc003a50 2024-09-13T11:54:59.044 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe5b6d640 1 --2- 172.21.15.100:0/301100211 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fcc003a50 0x7f0fe01cc770 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc00b9e0 tx=0x7f0fdc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:54:59.044 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fd67fc640 1 -- 172.21.15.100:0/301100211 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc043020 con 0x7f0fcc003a50 2024-09-13T11:54:59.044 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:54:59.044 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fd67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:54:59.044 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fd67fc640 10 monclient: dump: 2024-09-13T11:54:59.044 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:54:59.044 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.044 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.044 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.044 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:54:59.044 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.044 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:54:59.045 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:54:59.045 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:54:59.045 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fd67fc640 10 monclient: _finish_auth 0 2024-09-13T11:54:59.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fd67fc640 10 monclient: _check_auth_tickets 2024-09-13T11:54:59.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:54:59.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fd67fc640 1 -- 172.21.15.100:0/301100211 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fdc03f680 con 0x7f0fcc003a50 2024-09-13T11:54:59.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe7dd1640 5 monclient: authenticate success, global_id 4814 2024-09-13T11:54:59.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe7dd1640 10 monclient: _renew_subs 2024-09-13T11:54:59.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe7dd1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:54:59.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe7dd1640 1 -- 172.21.15.100:0/301100211 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fe00a1140 con 0x7f0fcc003a50 2024-09-13T11:54:59.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fd67fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:54:59.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fd67fc640 1 -- 172.21.15.100:0/301100211 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc00b5e0 con 0x7f0fcc003a50 2024-09-13T11:54:59.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:54:59.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fd67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:54:59.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fd67fc640 10 monclient: dump: 2024-09-13T11:54:59.046 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:54:59.046 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.046 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.046 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.046 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:54:59.046 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.046 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:54:59.046 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:54:59.046 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:54:59.046 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fd7fff640 4 set_mon_vals no callback set 2024-09-13T11:54:59.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe7dd1640 10 monclient: _renew_subs 2024-09-13T11:54:59.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe7dd1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:54:59.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.020+0000 7f0fe7dd1640 1 -- 172.21.15.100:0/301100211 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fe0128060 con 0x7f0fcc003a50 2024-09-13T11:54:59.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.021+0000 7f0fe7dd1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:54:59.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.021+0000 7f0fe7dd1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:54:59.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.021+0000 7f0fe7dd1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:54:59.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.021+0000 7f0fe7dd1640 1 -- 172.21.15.100:0/301100211 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fe00a2b30 con 0x7f0fcc003a50 2024-09-13T11:54:59.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.022+0000 7f0fd67fc640 1 -- 172.21.15.100:0/301100211 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f0fdc046020 con 0x7f0fcc003a50 2024-09-13T11:54:59.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.022+0000 7f0fd67fc640 1 --2- 172.21.15.100:0/301100211 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f0fc403dfd0 0x7f0fc4040490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.022+0000 7f0fd67fc640 1 -- 172.21.15.100:0/301100211 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(111..111 src has 1..111) ==== 4738+0+0 (secure 0 0 0) 0x7f0fdc073240 con 0x7f0fcc003a50 2024-09-13T11:54:59.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.022+0000 7f0fe536c640 1 --2- 172.21.15.100:0/301100211 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f0fc403dfd0 0x7f0fc4040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.022+0000 7f0fe536c640 10 monclient: get_auth_request con 0x7f0fc403dfd0 auth_method 0 2024-09-13T11:54:59.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.023+0000 7f0fe536c640 1 --2- 172.21.15.100:0/301100211 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f0fc403dfd0 0x7f0fc4040490 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0007fd0 tx=0x7f0fd0007b60 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:54:59.048 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.026+0000 7f0fd67fc640 1 -- 172.21.15.100:0/301100211 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f0fdc04d030 con 0x7f0fcc003a50 2024-09-13T11:54:59.048 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.026+0000 7f0fd67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:54:59.048 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.026+0000 7f0fd67fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:54:59.063 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36158]: starting. 2024-09-13T11:54:59.063 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36158]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-36156 2024-09-13T11:54:59.063 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36158]: shutting down. 2024-09-13T11:54:59.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.285+0000 7f0fe7dd1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:54:59.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.285+0000 7f0fe7dd1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:54:59.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.285+0000 7f0fe7dd1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:54:59.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.285+0000 7f0fe7dd1640 1 -- 172.21.15.100:0/301100211 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0fe006a0f0 con 0x7f0fcc003a50 2024-09-13T11:54:59.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.286+0000 7f0fd67fc640 1 -- 172.21.15.100:0/301100211 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f0fdc03c3a0 con 0x7f0fcc003a50 2024-09-13T11:54:59.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.286+0000 7f0fd67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:54:59.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.286+0000 7f0fd67fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:54:59.288 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:54:59.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.288+0000 7f0fe7dd1640 1 -- 172.21.15.100:0/301100211 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f0fc403dfd0 msgr2=0x7f0fc4040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.288+0000 7f0fe7dd1640 1 --2- 172.21.15.100:0/301100211 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f0fc403dfd0 0x7f0fc4040490 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0007fd0 tx=0x7f0fd0007b60 comp rx=0 tx=0).stop 2024-09-13T11:54:59.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.288+0000 7f0fe7dd1640 10 monclient: shutdown 2024-09-13T11:54:59.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.288+0000 7f0fe7dd1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:54:59.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.288+0000 7f0fe7dd1640 1 -- 172.21.15.100:0/301100211 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fcc003a50 msgr2=0x7f0fe01cc770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.288+0000 7f0fe7dd1640 1 --2- 172.21.15.100:0/301100211 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fcc003a50 0x7f0fe01cc770 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc00b9e0 tx=0x7f0fdc004730 comp rx=0 tx=0).stop 2024-09-13T11:54:59.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.288+0000 7f0fe7dd1640 1 -- 172.21.15.100:0/301100211 shutdown_connections 2024-09-13T11:54:59.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.288+0000 7f0fe7dd1640 1 --2- 172.21.15.100:0/301100211 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f0fc403dfd0 0x7f0fc4040490 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.288+0000 7f0fe7dd1640 1 --2- 172.21.15.100:0/301100211 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fe0124290 0x7f0fe0124740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.288+0000 7f0fe7dd1640 1 --2- 172.21.15.100:0/301100211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0fe0126860 0x7f0fe01cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.288+0000 7f0fe7dd1640 1 --2- 172.21.15.100:0/301100211 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fcc003a50 0x7f0fe01cc770 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.288+0000 7f0fe7dd1640 1 -- 172.21.15.100:0/301100211 >> 172.21.15.100:0/301100211 conn(0x7f0fe00aa580 msgr2=0x7f0fe00a89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:54:59.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.288+0000 7f0fe7dd1640 1 -- 172.21.15.100:0/301100211 shutdown_connections 2024-09-13T11:54:59.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.288+0000 7f0fe7dd1640 1 -- 172.21.15.100:0/301100211 wait complete. 2024-09-13T11:54:59.300 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-13T11:54:59.300 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-13T11:54:59.300 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-13T11:54:59.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.399+0000 7f63240ef640 10 monclient: get_monmap_and_config 2024-09-13T11:54:59.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.399+0000 7f63240ef640 10 monclient: build_initial_monmap 2024-09-13T11:54:59.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.399+0000 7f63240ef640 10 monclient: monmap: 2024-09-13T11:54:59.401 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:54:59.401 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.401 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:54:59.400597+0000 2024-09-13T11:54:59.401 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:54:59.400597+0000 2024-09-13T11:54:59.401 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:54:59.401 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.401 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:54:59.402 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:54:59.402 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:54:59.402 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:54:59.402 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:54:59.402 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.410 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.400+0000 7f63240ef640 1 Processor -- start 2024-09-13T11:54:59.411 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.400+0000 7f63240ef640 1 -- start start 2024-09-13T11:54:59.411 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.400+0000 7f63240ef640 10 monclient: init 2024-09-13T11:54:59.411 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.400+0000 7f63240ef640 10 monclient: _reopen_session rank -1 2024-09-13T11:54:59.411 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.400+0000 7f63240ef640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-13T11:54:59.411 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.400+0000 7f63240ef640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f631c06ea50 0x7f631c0a2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.411 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.400+0000 7f63240ef640 10 monclient(hunting): picked mon.noname-a con 0x7f631c06ea50 addr v2:172.21.15.100:3300/0 2024-09-13T11:54:59.411 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.400+0000 7f63240ef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f631c06ee30 addr v1:172.21.15.100:6789/0 2024-09-13T11:54:59.411 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.400+0000 7f63240ef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f631c13c9f0 addr v1:172.21.15.122:6789/0 2024-09-13T11:54:59.411 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.400+0000 7f63240ef640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f631c0a1590 con 0x7f631c06ee30 2024-09-13T11:54:59.411 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.400+0000 7f63240ef640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f631c0a1870 con 0x7f631c13c9f0 2024-09-13T11:54:59.411 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.400+0000 7f63240ef640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.411 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.400+0000 7f63240ef640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f631c0a1b50 con 0x7f631c06ea50 2024-09-13T11:54:59.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.400+0000 7f63240ef640 10 monclient(hunting): _renew_subs 2024-09-13T11:54:59.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.400+0000 7f63240ef640 10 monclient(hunting): authenticate will time out at 879.993986s 2024-09-13T11:54:59.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.401+0000 7f6321e8b640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f631c06ea50 0x7f631c0a2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.401+0000 7f6321e8b640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f631c06ea50 0x7f631c0a2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53374/0 (socket says 172.21.15.100:53374) 2024-09-13T11:54:59.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.401+0000 7f6321e8b640 1 -- 172.21.15.100:0/2298490104 learned_addr learned my addr 172.21.15.100:0/2298490104 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:54:59.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.401+0000 7f6321e8b640 10 monclient(hunting): get_auth_request con 0x7f631c06ea50 auth_method 0 2024-09-13T11:54:59.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.401+0000 7f6321e8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.401+0000 7f6321e8b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.401+0000 7f6321e8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.401+0000 7f6321e8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:54:59.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.401+0000 7f6321e8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:54:59.412 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.401+0000 7f6321e8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:54:59.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.401+0000 7f6321e8b640 10 monclient(hunting): handle_auth_done global_id 14097 payload 274 2024-09-13T11:54:59.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.401+0000 7f6321e8b640 1 -- 172.21.15.100:0/2298490104 >> v1:172.21.15.122:6789/0 conn(0x7f631c13c9f0 legacy=0x7f631c13ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:54:59.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.401+0000 7f6321e8b640 1 -- 172.21.15.100:0/2298490104 >> v1:172.21.15.100:6789/0 conn(0x7f631c06ee30 legacy=0x7f631c06f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.401+0000 7f6321e8b640 10 monclient: _finish_hunting 0 2024-09-13T11:54:59.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.401+0000 7f6321e8b640 1 monclient: found mon.noname-a 2024-09-13T11:54:59.413 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.401+0000 7f6321e8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:54:59.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.401+0000 7f6321e8b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:54:59.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.401+0000 7f6321e8b640 1 -- 172.21.15.100:0/2298490104 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f631c06fca0 con 0x7f631c06ea50 2024-09-13T11:54:59.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f6321e8b640 1 --2- 172.21.15.100:0/2298490104 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c06ea50 0x7f631c0a2860 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6318009870 tx=0x7f631802a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7afa6f89b37b6de server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:54:59.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f6320e89640 1 -- 172.21.15.100:0/2298490104 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6318037070 con 0x7f631c06ea50 2024-09-13T11:54:59.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f6320e89640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:54:59.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f6320e89640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:54:59.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f6320e89640 10 monclient: dump: 2024-09-13T11:54:59.414 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:54:59.415 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.415 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.415 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.415 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:54:59.415 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.415 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:54:59.415 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:54:59.415 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:54:59.415 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.415 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f6320e89640 10 monclient: _finish_auth 0 2024-09-13T11:54:59.415 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f6320e89640 10 monclient: _check_auth_tickets 2024-09-13T11:54:59.415 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f6320e89640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:54:59.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f6320e89640 1 -- 172.21.15.100:0/2298490104 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6318032440 con 0x7f631c06ea50 2024-09-13T11:54:59.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f63240ef640 5 monclient: authenticate success, global_id 14097 2024-09-13T11:54:59.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f63240ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:54:59.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f6320e89640 10 monclient: handle_config config(0 keys) 2024-09-13T11:54:59.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f63240ef640 10 monclient: get_monmap_and_config success 2024-09-13T11:54:59.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f6320e89640 1 -- 172.21.15.100:0/2298490104 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6318032810 con 0x7f631c06ea50 2024-09-13T11:54:59.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f6320e89640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:54:59.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f63240ef640 4 set_mon_vals no callback set 2024-09-13T11:54:59.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f6320e89640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:54:59.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f6320e89640 10 monclient: dump: 2024-09-13T11:54:59.417 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:54:59.417 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.417 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.417 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.417 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:54:59.417 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.417 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:54:59.417 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:54:59.417 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:54:59.417 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.418 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f63240ef640 10 monclient: shutdown 2024-09-13T11:54:59.418 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f63240ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:54:59.418 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f63240ef640 1 -- 172.21.15.100:0/2298490104 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c06ea50 msgr2=0x7f631c0a2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f63240ef640 1 --2- 172.21.15.100:0/2298490104 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c06ea50 0x7f631c0a2860 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6318009870 tx=0x7f631802a130 comp rx=0 tx=0).stop 2024-09-13T11:54:59.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f63240ef640 1 -- 172.21.15.100:0/2298490104 shutdown_connections 2024-09-13T11:54:59.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f63240ef640 1 --2- 172.21.15.100:0/2298490104 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c06ea50 0x7f631c0a2860 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f63240ef640 1 -- 172.21.15.100:0/2298490104 >> 172.21.15.100:0/2298490104 conn(0x7f631c0a68e0 msgr2=0x7f631c0a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:54:59.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f63240ef640 1 -- 172.21.15.100:0/2298490104 shutdown_connections 2024-09-13T11:54:59.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.402+0000 7f63240ef640 1 -- 172.21.15.100:0/2298490104 wait complete. 2024-09-13T11:54:59.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.403+0000 7f63240ef640 10 monclient: build_initial_monmap 2024-09-13T11:54:59.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.403+0000 7f63240ef640 10 monclient: monmap: 2024-09-13T11:54:59.419 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:54:59.420 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:54:59.420 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:54:59.420 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:54:59.420 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:54:59.420 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.420 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:54:59.420 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:54:59.421 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:54:59.421 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.403+0000 7f63240ef640 1 Processor -- start 2024-09-13T11:54:59.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.403+0000 7f63240ef640 1 -- start start 2024-09-13T11:54:59.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.403+0000 7f63240ef640 10 monclient: init 2024-09-13T11:54:59.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63240ef640 10 monclient: _reopen_session rank -1 2024-09-13T11:54:59.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63240ef640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:54:59.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63240ef640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c06ea50 0x7f631c137010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63240ef640 10 monclient(hunting): picked mon.noname-a con 0x7f631c06ea50 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:54:59.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63240ef640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f631c06ee30 0x7f631c137550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63240ef640 10 monclient(hunting): picked mon.noname-b con 0x7f631c06ee30 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:54:59.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63240ef640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f631c13c9f0 0x7f631c09d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63240ef640 10 monclient(hunting): picked mon.noname-c con 0x7f631c13c9f0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:54:59.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63240ef640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63240ef640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f631c140680 con 0x7f631c06ea50 2024-09-13T11:54:59.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63240ef640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63240ef640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f631c140500 con 0x7f631c06ee30 2024-09-13T11:54:59.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63240ef640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63240ef640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f631c140800 con 0x7f631c13c9f0 2024-09-13T11:54:59.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63240ef640 10 monclient(hunting): _renew_subs 2024-09-13T11:54:59.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63240ef640 10 monclient(hunting): authenticate will time out at 879.997236s 2024-09-13T11:54:59.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f6321e8b640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c06ea50 0x7f631c137010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f6321e8b640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c06ea50 0x7f631c137010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53380/0 (socket says 172.21.15.100:53380) 2024-09-13T11:54:59.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f6321e8b640 1 -- 172.21.15.100:0/3012949044 learned_addr learned my addr 172.21.15.100:0/3012949044 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:54:59.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f6321e8b640 10 monclient(hunting): get_auth_request con 0x7f631c06ea50 auth_method 0 2024-09-13T11:54:59.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f6321e8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.423 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f6321e8b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.423 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f6321e8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.423 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f632268c640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f631c13c9f0 0x7f631c09d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.423 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f632268c640 10 monclient(hunting): get_auth_request con 0x7f631c13c9f0 auth_method 0 2024-09-13T11:54:59.423 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f632268c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.423 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f632268c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.423 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f632268c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.423 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f6321e8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:54:59.423 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f6321e8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:54:59.423 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f6321e8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:54:59.423 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f632268c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:54:59.423 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f632268c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:54:59.423 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f632268c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:54:59.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f6321e8b640 10 monclient(hunting): handle_auth_done global_id 14103 payload 824 2024-09-13T11:54:59.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f6321e8b640 1 -- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f631c13c9f0 msgr2=0x7f631c09d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f6321e8b640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f631c13c9f0 0x7f631c09d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f6321e8b640 1 -- 172.21.15.100:0/3012949044 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f631c06ee30 msgr2=0x7f631c137550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f6321e8b640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f631c06ee30 0x7f631c137550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f6321e8b640 10 monclient: _finish_hunting 0 2024-09-13T11:54:59.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f6321e8b640 1 monclient: found mon.noname-a 2024-09-13T11:54:59.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f6321e8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:54:59.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f6321e8b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:54:59.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f6321e8b640 1 -- 172.21.15.100:0/3012949044 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f631c120430 con 0x7f631c06ea50 2024-09-13T11:54:59.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f6321e8b640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c06ea50 0x7f631c137010 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6318009870 tx=0x7f63180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:54:59.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63127fc640 1 -- 172.21.15.100:0/3012949044 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6318041070 con 0x7f631c06ea50 2024-09-13T11:54:59.425 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:54:59.425 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63127fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:54:59.425 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63127fc640 10 monclient: dump: 2024-09-13T11:54:59.425 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:54:59.425 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.425 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.425 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.425 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:54:59.425 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.425 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:54:59.425 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:54:59.426 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:54:59.426 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63127fc640 10 monclient: _finish_auth 0 2024-09-13T11:54:59.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63127fc640 10 monclient: _check_auth_tickets 2024-09-13T11:54:59.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:54:59.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63127fc640 1 -- 172.21.15.100:0/3012949044 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63180308f0 con 0x7f631c06ea50 2024-09-13T11:54:59.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63240ef640 5 monclient: authenticate success, global_id 14103 2024-09-13T11:54:59.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63240ef640 10 monclient: _renew_subs 2024-09-13T11:54:59.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63240ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:54:59.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63240ef640 1 -- 172.21.15.100:0/3012949044 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f631c120990 con 0x7f631c06ea50 2024-09-13T11:54:59.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63127fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:54:59.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63127fc640 1 -- 172.21.15.100:0/3012949044 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6318037040 con 0x7f631c06ea50 2024-09-13T11:54:59.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:54:59.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:54:59.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63127fc640 10 monclient: dump: 2024-09-13T11:54:59.427 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:54:59.427 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.427 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.427 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.427 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:54:59.427 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.427 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:54:59.427 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:54:59.427 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:54:59.427 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63240ef640 10 monclient: _renew_subs 2024-09-13T11:54:59.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63240ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:54:59.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f63240ef640 1 -- 172.21.15.100:0/3012949044 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f631c1206b0 con 0x7f631c06ea50 2024-09-13T11:54:59.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.404+0000 7f6313fff640 4 set_mon_vals no callback set 2024-09-13T11:54:59.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.405+0000 7f63240ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:54:59.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.405+0000 7f63240ef640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c09df30 0x7f631c09e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.405+0000 7f63240ef640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.405+0000 7f63240ef640 1 -- 172.21.15.100:0/3012949044 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f631c13f9b0 con 0x7f631c09df30 2024-09-13T11:54:59.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.405+0000 7f63127fc640 1 -- 172.21.15.100:0/3012949044 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f6318044070 con 0x7f631c06ea50 2024-09-13T11:54:59.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.405+0000 7f632168a640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c09df30 0x7f631c09e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.405+0000 7f632168a640 10 monclient: get_auth_request con 0x7f631c09df30 auth_method 0 2024-09-13T11:54:59.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.405+0000 7f632168a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.405+0000 7f632168a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.405+0000 7f632168a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.405+0000 7f632168a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:54:59.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.405+0000 7f632168a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:54:59.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.406+0000 7f632168a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:54:59.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.406+0000 7f632168a640 10 monclient(hunting): handle_auth_done global_id 14106 payload 824 2024-09-13T11:54:59.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.406+0000 7f632168a640 1 -- 172.21.15.100:0/3012949044 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f631c0a2860 con 0x7f631c09df30 2024-09-13T11:54:59.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.406+0000 7f63127fc640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f630003dfd0 0x7f6300040490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.406+0000 7f63127fc640 1 -- 172.21.15.100:0/3012949044 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 4738+0+0 (secure 0 0 0) 0x7f6318073430 con 0x7f631c06ea50 2024-09-13T11:54:59.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.406+0000 7f632168a640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c09df30 0x7f631c09e310 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f631c09fc50 tx=0x7f630c00c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:54:59.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.406+0000 7f63127fc640 1 -- 172.21.15.100:0/3012949044 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630c017070 con 0x7f631c09df30 2024-09-13T11:54:59.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.406+0000 7f63127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:54:59.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.406+0000 7f63127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:54:59.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.406+0000 7f63127fc640 10 monclient: dump: 2024-09-13T11:54:59.429 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:54:59.430 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.430 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.430 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.430 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:54:59.430 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.430 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:54:59.430 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:54:59.430 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:54:59.430 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.430 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.406+0000 7f632268c640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f630003dfd0 0x7f6300040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.430 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.406+0000 7f632268c640 10 monclient: get_auth_request con 0x7f630003dfd0 auth_method 0 2024-09-13T11:54:59.430 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.406+0000 7f63127fc640 1 -- 172.21.15.100:0/3012949044 <== mon.0 v2:172.21.15.100:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f630c0052d0 con 0x7f631c09df30 2024-09-13T11:54:59.430 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.406+0000 7f63127fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:54:59.430 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.406+0000 7f63127fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:54:59.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.406+0000 7f63127fc640 1 -- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c09df30 msgr2=0x7f631c09e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.406+0000 7f63127fc640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c09df30 0x7f631c09e310 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f631c09fc50 tx=0x7f630c00c780 comp rx=0 tx=0).stop 2024-09-13T11:54:59.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.406+0000 7f63127fc640 1 -- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c09df30 msgr2=0x7f631c09e310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:54:59.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.406+0000 7f63127fc640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c09df30 0x7f631c09e310 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.406+0000 7f632268c640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f630003dfd0 0x7f6300040490 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f631c0a6590 tx=0x7f6314003d10 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:54:59.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.412+0000 7f63240ef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:54:59.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.412+0000 7f63240ef640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c09e850 0x7f631c13d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.412+0000 7f63240ef640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.412+0000 7f63240ef640 1 -- 172.21.15.100:0/3012949044 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f631c069100 con 0x7f631c09e850 2024-09-13T11:54:59.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.412+0000 7f632168a640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c09e850 0x7f631c13d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.412+0000 7f632168a640 10 monclient: get_auth_request con 0x7f631c09e850 auth_method 0 2024-09-13T11:54:59.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.412+0000 7f632168a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.412+0000 7f632168a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.412+0000 7f632168a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f632168a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:54:59.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f632168a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:54:59.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f632168a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:54:59.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f632168a640 10 monclient(hunting): handle_auth_done global_id 14109 payload 824 2024-09-13T11:54:59.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f632168a640 1 -- 172.21.15.100:0/3012949044 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f631c09fc80 con 0x7f631c09e850 2024-09-13T11:54:59.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f632168a640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c09e850 0x7f631c13d770 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f631c09fc50 tx=0x7f630c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:54:59.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f632168a640 1 -- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c09e850 msgr2=0x7f631c13d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-13T11:54:59.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f632168a640 1 -- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c09e850 msgr2=0x7f631c13d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T11:54:59.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f632168a640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c09e850 0x7f631c13d770 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f631c09fc50 tx=0x7f630c000ed0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:54:59.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f632168a640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c09e850 0x7f631c13d770 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f631c09fc50 tx=0x7f630c000ed0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f63127fc640 1 -- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c09e850 msgr2=0x7f631c13d770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:54:59.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f63127fc640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c09e850 0x7f631c13d770 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f63127fc640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6300044680 0x7f6300046a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f63127fc640 1 -- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c09e850 msgr2=0x7f631c13d770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:54:59.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f63127fc640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c09e850 0x7f631c13d770 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f63127fc640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f63127fc640 1 -- 172.21.15.100:0/3012949044 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f6300047080 con 0x7f6300044680 2024-09-13T11:54:59.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f632168a640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6300044680 0x7f6300046a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f632168a640 10 monclient: get_auth_request con 0x7f6300044680 auth_method 0 2024-09-13T11:54:59.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f632168a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f632168a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f632168a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f632168a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:54:59.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f632168a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:54:59.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f632168a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:54:59.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f632168a640 10 monclient(hunting): handle_auth_done global_id 14112 payload 824 2024-09-13T11:54:59.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f632168a640 1 -- 172.21.15.100:0/3012949044 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f630c0052d0 con 0x7f6300044680 2024-09-13T11:54:59.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f632168a640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6300044680 0x7f6300046a70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f631c069280 tx=0x7f630c00c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:54:59.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.413+0000 7f63127fc640 1 -- 172.21.15.100:0/3012949044 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630c030030 con 0x7f6300044680 2024-09-13T11:54:59.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.414+0000 7f63127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:54:59.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.414+0000 7f63127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:54:59.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.414+0000 7f63127fc640 10 monclient: dump: 2024-09-13T11:54:59.437 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:54:59.437 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.437 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.437 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.437 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:54:59.438 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.438 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:54:59.438 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:54:59.438 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:54:59.438 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.414+0000 7f63127fc640 1 -- 172.21.15.100:0/3012949044 <== mon.0 v2:172.21.15.100:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f630c01a5f0 con 0x7f6300044680 2024-09-13T11:54:59.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.414+0000 7f63127fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:54:59.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.414+0000 7f63127fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:54:59.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.414+0000 7f63127fc640 1 -- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6300044680 msgr2=0x7f6300046a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.414+0000 7f63127fc640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6300044680 0x7f6300046a70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f631c069280 tx=0x7f630c00c780 comp rx=0 tx=0).stop 2024-09-13T11:54:59.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.414+0000 7f63127fc640 1 -- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6300044680 msgr2=0x7f6300046a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:54:59.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.414+0000 7f63127fc640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6300044680 0x7f6300046a70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.414+0000 7f632268c640 1 -- 172.21.15.100:0/3012949044 reap_dead start 2024-09-13T11:54:59.439 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:54:59.439 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:54:59.439 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:54:59.439 INFO:teuthology.orchestra.run.smithi100.stdout: "state": "leader", 2024-09-13T11:54:59.439 INFO:teuthology.orchestra.run.smithi100.stdout: "election_epoch": 16, 2024-09-13T11:54:59.439 INFO:teuthology.orchestra.run.smithi100.stdout: "uptime": 413, 2024-09-13T11:54:59.439 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum": [ 2024-09-13T11:54:59.439 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-13T11:54:59.439 INFO:teuthology.orchestra.run.smithi100.stdout: 1, 2024-09-13T11:54:59.439 INFO:teuthology.orchestra.run.smithi100.stdout: 2 2024-09-13T11:54:59.439 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:54:59.440 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_age": 0, 2024-09-13T11:54:59.440 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:54:59.440 INFO:teuthology.orchestra.run.smithi100.stdout: "required_con": "2450521708007866372", 2024-09-13T11:54:59.440 INFO:teuthology.orchestra.run.smithi100.stdout: "required_mon": [ 2024-09-13T11:54:59.440 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:54:59.440 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:54:59.440 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:54:59.440 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:54:59.440 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:54:59.440 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:54:59.440 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:54:59.440 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:54:59.440 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:54:59.441 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:54:59.441 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:54:59.441 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:54:59.441 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:54:59.441 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_mon": [ 2024-09-13T11:54:59.441 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:54:59.441 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:54:59.441 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:54:59.441 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:54:59.441 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:54:59.441 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:54:59.441 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:54:59.441 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:54:59.442 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:54:59.442 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:54:59.442 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:54:59.442 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:54:59.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.442 INFO:teuthology.orchestra.run.smithi100.stdout: "outside_quorum": [], 2024-09-13T11:54:59.442 INFO:teuthology.orchestra.run.smithi100.stdout: "extra_probe_peers": [], 2024-09-13T11:54:59.442 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_provider": [], 2024-09-13T11:54:59.442 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": { 2024-09-13T11:54:59.442 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-09-13T11:54:59.442 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "0b1f4b4d-fad4-4dd3-86ac-a286d80846c0", 2024-09-13T11:54:59.442 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:54:59.442 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:54:59.442 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release": 19, 2024-09-13T11:54:59.443 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release_name": "squid", 2024-09-13T11:54:59.443 INFO:teuthology.orchestra.run.smithi100.stdout: "election_strategy": 1, 2024-09-13T11:54:59.443 INFO:teuthology.orchestra.run.smithi100.stdout: "disallowed_leaders: ": "", 2024-09-13T11:54:59.443 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false, 2024-09-13T11:54:59.443 INFO:teuthology.orchestra.run.smithi100.stdout: "tiebreaker_mon": "", 2024-09-13T11:54:59.443 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_ranks: ": "", 2024-09-13T11:54:59.443 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:54:59.443 INFO:teuthology.orchestra.run.smithi100.stdout: "persistent": [ 2024-09-13T11:54:59.443 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:54:59.443 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:54:59.443 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:54:59.443 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:54:59.443 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:54:59.444 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:54:59.444 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:54:59.444 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:54:59.444 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:54:59.444 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:54:59.444 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:54:59.444 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:54:59.444 INFO:teuthology.orchestra.run.smithi100.stdout: "optional": [] 2024-09-13T11:54:59.444 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.444 INFO:teuthology.orchestra.run.smithi100.stdout: "mons": [ 2024-09-13T11:54:59.444 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.444 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:54:59.445 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:54:59.445 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:54:59.445 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:54:59.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.445 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:54:59.445 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3300", 2024-09-13T11:54:59.445 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:54:59.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.445 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:54:59.445 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789", 2024-09-13T11:54:59.445 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:54:59.445 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:54:59.445 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:54:59.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.446 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789/0", 2024-09-13T11:54:59.446 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-13T11:54:59.446 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:54:59.446 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:54:59.446 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:54:59.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.446 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:54:59.446 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:54:59.446 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:54:59.446 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:54:59.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.447 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:54:59.447 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:3300", 2024-09-13T11:54:59.447 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:54:59.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.447 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:54:59.447 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789", 2024-09-13T11:54:59.447 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:54:59.447 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:54:59.447 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:54:59.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.447 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789/0", 2024-09-13T11:54:59.447 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.122:6789/0", 2024-09-13T11:54:59.447 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:54:59.448 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:54:59.448 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:54:59.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.448 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:54:59.448 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:54:59.448 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:54:59.448 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:54:59.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.448 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:54:59.448 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3301", 2024-09-13T11:54:59.448 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:54:59.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.449 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:54:59.449 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790", 2024-09-13T11:54:59.449 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:54:59.449 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:54:59.449 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:54:59.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.449 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790/0", 2024-09-13T11:54:59.449 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-13T11:54:59.449 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:54:59.449 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:54:59.449 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:54:59.449 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:54:59.449 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:54:59.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.450 INFO:teuthology.orchestra.run.smithi100.stdout: "feature_map": { 2024-09-13T11:54:59.450 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": [ 2024-09-13T11:54:59.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.450 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:54:59.450 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:54:59.450 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:54:59.450 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:54:59.450 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:54:59.450 INFO:teuthology.orchestra.run.smithi100.stdout: "client": [ 2024-09-13T11:54:59.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.450 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:54:59.450 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:54:59.451 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 2 2024-09-13T11:54:59.451 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:54:59.451 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:54:59.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.451 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false 2024-09-13T11:54:59.451 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:54:59.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.414+0000 7f63240ef640 1 -- 172.21.15.100:0/3012949044 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f630003dfd0 msgr2=0x7f6300040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.414+0000 7f63240ef640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f630003dfd0 0x7f6300040490 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f631c0a6590 tx=0x7f6314003d10 comp rx=0 tx=0).stop 2024-09-13T11:54:59.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.414+0000 7f63240ef640 10 monclient: shutdown 2024-09-13T11:54:59.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.414+0000 7f63240ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:54:59.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.414+0000 7f63240ef640 1 -- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c06ea50 msgr2=0x7f631c137010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.414+0000 7f63240ef640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c06ea50 0x7f631c137010 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6318009870 tx=0x7f63180046c0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.414+0000 7f63240ef640 1 -- 172.21.15.100:0/3012949044 shutdown_connections 2024-09-13T11:54:59.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.414+0000 7f63240ef640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f630003dfd0 0x7f6300040490 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.414+0000 7f63240ef640 1 --2- 172.21.15.100:0/3012949044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f631c06ea50 0x7f631c137010 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.414+0000 7f63240ef640 1 -- 172.21.15.100:0/3012949044 >> 172.21.15.100:0/3012949044 conn(0x7f631c0a68e0 msgr2=0x7f631c13bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:54:59.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.414+0000 7f63240ef640 1 -- 172.21.15.100:0/3012949044 shutdown_connections 2024-09-13T11:54:59.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.414+0000 7f63240ef640 1 -- 172.21.15.100:0/3012949044 wait complete. 2024-09-13T11:54:59.453 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T11:54:59.454 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-13T11:54:59.454 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T11:54:59.552 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.550+0000 7f863054b640 10 monclient: get_monmap_and_config 2024-09-13T11:54:59.552 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.550+0000 7f863054b640 10 monclient: build_initial_monmap 2024-09-13T11:54:59.552 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.551+0000 7f863054b640 10 monclient: monmap: 2024-09-13T11:54:59.552 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:54:59.552 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.552 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:54:59.551854+0000 2024-09-13T11:54:59.552 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:54:59.551854+0000 2024-09-13T11:54:59.553 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:54:59.553 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.553 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:54:59.553 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:54:59.553 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:54:59.553 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:54:59.553 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:54:59.553 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.551+0000 7f863054b640 1 Processor -- start 2024-09-13T11:54:59.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.551+0000 7f863054b640 1 -- start start 2024-09-13T11:54:59.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.551+0000 7f863054b640 10 monclient: init 2024-09-13T11:54:59.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f863054b640 10 monclient: _reopen_session rank -1 2024-09-13T11:54:59.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f863054b640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-13T11:54:59.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f863054b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f86280a37d0 addr v1:172.21.15.100:6789/0 2024-09-13T11:54:59.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f863054b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f862806eeb0 addr v1:172.21.15.122:6789/0 2024-09-13T11:54:59.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f863054b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86280a42c0 0x7f8628140350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f863054b640 10 monclient(hunting): picked mon.noname-b con 0x7f86280a42c0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:54:59.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f863054b640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f862806e950 con 0x7f86280a37d0 2024-09-13T11:54:59.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f863054b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86280a5240 con 0x7f862806eeb0 2024-09-13T11:54:59.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f863054b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f863054b640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f862809fde0 con 0x7f86280a42c0 2024-09-13T11:54:59.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f863054b640 10 monclient(hunting): _renew_subs 2024-09-13T11:54:59.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f863054b640 10 monclient(hunting): authenticate will time out at 880.145375s 2024-09-13T11:54:59.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f862e2e7640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f86280a37d0 0x7f86280a3bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:48234/0 (socket says 172.21.15.100:48234) 2024-09-13T11:54:59.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f862e2e7640 1 -- 172.21.15.100:0/1245293311 learned_addr learned my addr 172.21.15.100:0/1245293311 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:54:59.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f862eae8640 1 --2- 172.21.15.100:0/1245293311 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86280a42c0 0x7f8628140350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f862eae8640 10 monclient(hunting): get_auth_request con 0x7f86280a42c0 auth_method 0 2024-09-13T11:54:59.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f862eae8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f862eae8640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f862eae8640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f862eae8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:54:59.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f862eae8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:54:59.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f862eae8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:54:59.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f862d2e5640 1 -- 172.21.15.100:0/1245293311 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 324503925 0 0) 0x7f862806e950 con 0x7f86280a37d0 2024-09-13T11:54:59.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f862d2e5640 10 monclient(hunting): _negotiate 2024-09-13T11:54:59.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f862d2e5640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f862d2e5640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f862d2e5640 10 monclient(hunting): my global_id is 14115 2024-09-13T11:54:59.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f862d2e5640 1 -- 172.21.15.100:0/1245293311 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8614003520 con 0x7f86280a37d0 2024-09-13T11:54:59.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f862eae8640 10 monclient(hunting): handle_auth_done global_id 4832 payload 274 2024-09-13T11:54:59.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f862eae8640 1 -- 172.21.15.100:0/1245293311 >> v1:172.21.15.122:6789/0 conn(0x7f862806eeb0 legacy=0x7f862813cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f862eae8640 1 -- 172.21.15.100:0/1245293311 >> v1:172.21.15.100:6789/0 conn(0x7f86280a37d0 legacy=0x7f86280a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f862eae8640 10 monclient: _finish_hunting 0 2024-09-13T11:54:59.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f862eae8640 1 monclient: found mon.noname-b 2024-09-13T11:54:59.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f862eae8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:54:59.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f862eae8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:54:59.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.552+0000 7f862eae8640 1 -- 172.21.15.100:0/1245293311 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f862809ec50 con 0x7f86280a42c0 2024-09-13T11:54:59.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.553+0000 7f862eae8640 1 --2- 172.21.15.100:0/1245293311 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86280a42c0 0x7f8628140350 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8620009dc0 tx=0x7f862002a5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2fafc158b8f99ad server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:54:59.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.553+0000 7f862d2e5640 1 -- 172.21.15.100:0/1245293311 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8620007740 con 0x7f86280a42c0 2024-09-13T11:54:59.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.553+0000 7f862d2e5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:54:59.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.553+0000 7f862d2e5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:54:59.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.553+0000 7f862d2e5640 10 monclient: dump: 2024-09-13T11:54:59.567 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:54:59.567 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.567 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.567 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.567 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:54:59.567 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.567 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:54:59.567 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:54:59.567 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:54:59.567 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.553+0000 7f862d2e5640 10 monclient: _finish_auth 0 2024-09-13T11:54:59.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.553+0000 7f862d2e5640 10 monclient: _check_auth_tickets 2024-09-13T11:54:59.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.553+0000 7f862d2e5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:54:59.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.553+0000 7f862d2e5640 1 -- 172.21.15.100:0/1245293311 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86200078e0 con 0x7f86280a42c0 2024-09-13T11:54:59.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.553+0000 7f863054b640 5 monclient: authenticate success, global_id 4832 2024-09-13T11:54:59.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.553+0000 7f863054b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:54:59.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.553+0000 7f862d2e5640 10 monclient: handle_config config(0 keys) 2024-09-13T11:54:59.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.553+0000 7f863054b640 10 monclient: get_monmap_and_config success 2024-09-13T11:54:59.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.553+0000 7f862d2e5640 1 -- 172.21.15.100:0/1245293311 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8620007cb0 con 0x7f86280a42c0 2024-09-13T11:54:59.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.553+0000 7f863054b640 4 set_mon_vals no callback set 2024-09-13T11:54:59.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.553+0000 7f862d2e5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:54:59.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.553+0000 7f862d2e5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:54:59.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.553+0000 7f862d2e5640 10 monclient: dump: 2024-09-13T11:54:59.568 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:54:59.568 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.568 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.569 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.569 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:54:59.569 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.569 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:54:59.569 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:54:59.569 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:54:59.569 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.554+0000 7f863054b640 10 monclient: shutdown 2024-09-13T11:54:59.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.554+0000 7f863054b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:54:59.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.554+0000 7f863054b640 1 -- 172.21.15.100:0/1245293311 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86280a42c0 msgr2=0x7f8628140350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.554+0000 7f863054b640 1 --2- 172.21.15.100:0/1245293311 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86280a42c0 0x7f8628140350 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8620009dc0 tx=0x7f862002a5b0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.554+0000 7f863054b640 1 -- 172.21.15.100:0/1245293311 shutdown_connections 2024-09-13T11:54:59.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.554+0000 7f863054b640 1 --2- 172.21.15.100:0/1245293311 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86280a42c0 0x7f8628140350 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.554+0000 7f863054b640 1 -- 172.21.15.100:0/1245293311 >> 172.21.15.100:0/1245293311 conn(0x7f86280a83f0 msgr2=0x7f86280a48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:54:59.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.554+0000 7f863054b640 1 -- 172.21.15.100:0/1245293311 shutdown_connections 2024-09-13T11:54:59.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.554+0000 7f863054b640 1 -- 172.21.15.100:0/1245293311 wait complete. 2024-09-13T11:54:59.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.554+0000 7f863054b640 10 monclient: build_initial_monmap 2024-09-13T11:54:59.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.554+0000 7f863054b640 10 monclient: monmap: 2024-09-13T11:54:59.570 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:54:59.570 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:54:59.570 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:54:59.570 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:54:59.571 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:54:59.571 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.571 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:54:59.571 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:54:59.571 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:54:59.571 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f863054b640 1 Processor -- start 2024-09-13T11:54:59.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f863054b640 1 -- start start 2024-09-13T11:54:59.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f863054b640 10 monclient: init 2024-09-13T11:54:59.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f863054b640 10 monclient: _reopen_session rank -1 2024-09-13T11:54:59.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f863054b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:54:59.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f863054b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f862806eeb0 0x7f862813a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f863054b640 10 monclient(hunting): picked mon.noname-b con 0x7f862806eeb0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:54:59.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f863054b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86280a37d0 0x7f862813ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f863054b640 10 monclient(hunting): picked mon.noname-c con 0x7f86280a37d0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:54:59.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f863054b640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86280a42c0 0x7f862813b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f863054b640 10 monclient(hunting): picked mon.noname-a con 0x7f86280a42c0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:54:59.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f863054b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f863054b640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f8628140aa0 con 0x7f86280a42c0 2024-09-13T11:54:59.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f863054b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f863054b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f8628140c20 con 0x7f862806eeb0 2024-09-13T11:54:59.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f863054b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f863054b640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f8628140da0 con 0x7f86280a37d0 2024-09-13T11:54:59.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f863054b640 10 monclient(hunting): _renew_subs 2024-09-13T11:54:59.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f863054b640 10 monclient(hunting): authenticate will time out at 880.148624s 2024-09-13T11:54:59.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862dae6640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86280a37d0 0x7f862813ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862eae8640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86280a42c0 0x7f862813b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862dae6640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86280a37d0 0x7f862813ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:36870/0 (socket says 172.21.15.100:36870) 2024-09-13T11:54:59.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862dae6640 1 -- 172.21.15.100:0/2617049741 learned_addr learned my addr 172.21.15.100:0/2617049741 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:54:59.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862dae6640 10 monclient(hunting): get_auth_request con 0x7f86280a37d0 auth_method 0 2024-09-13T11:54:59.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862dae6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862dae6640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862dae6640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862eae8640 10 monclient(hunting): get_auth_request con 0x7f86280a42c0 auth_method 0 2024-09-13T11:54:59.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862eae8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862eae8640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862eae8640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862e2e7640 1 --2- 172.21.15.100:0/2617049741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f862806eeb0 0x7f862813a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862e2e7640 10 monclient(hunting): get_auth_request con 0x7f862806eeb0 auth_method 0 2024-09-13T11:54:59.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862e2e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862e2e7640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862e2e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862dae6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:54:59.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862dae6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:54:59.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862dae6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:54:59.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862eae8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:54:59.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862eae8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:54:59.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862eae8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:54:59.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862e2e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:54:59.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862e2e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:54:59.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862e2e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:54:59.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862dae6640 10 monclient(hunting): handle_auth_done global_id 4835 payload 824 2024-09-13T11:54:59.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862dae6640 1 -- 172.21.15.100:0/2617049741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f862806eeb0 msgr2=0x7f862813a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.555+0000 7f862dae6640 1 --2- 172.21.15.100:0/2617049741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f862806eeb0 0x7f862813a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f862dae6640 1 -- 172.21.15.100:0/2617049741 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86280a42c0 msgr2=0x7f862813b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f862dae6640 1 --2- 172.21.15.100:0/2617049741 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86280a42c0 0x7f862813b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f862dae6640 10 monclient: _finish_hunting 0 2024-09-13T11:54:59.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f862dae6640 1 monclient: found mon.noname-c 2024-09-13T11:54:59.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f862dae6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:54:59.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f862dae6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:54:59.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f862dae6640 1 -- 172.21.15.100:0/2617049741 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86280a61a0 con 0x7f86280a37d0 2024-09-13T11:54:59.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f862eae8640 1 --2- 172.21.15.100:0/2617049741 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86280a42c0 0x7f862813b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:54:59.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f862dae6640 1 --2- 172.21.15.100:0/2617049741 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86280a37d0 0x7f862813ae20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f861800c450 tx=0x7f861800c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:54:59.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f861effd640 1 -- 172.21.15.100:0/2617049741 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86180052d0 con 0x7f86280a37d0 2024-09-13T11:54:59.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f861effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:54:59.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f861effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:54:59.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f861effd640 10 monclient: dump: 2024-09-13T11:54:59.576 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:54:59.576 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.576 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.577 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.577 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:54:59.577 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.577 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:54:59.577 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:54:59.577 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:54:59.577 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f861effd640 10 monclient: _finish_auth 0 2024-09-13T11:54:59.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f861effd640 10 monclient: _check_auth_tickets 2024-09-13T11:54:59.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f861effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:54:59.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f861effd640 1 -- 172.21.15.100:0/2617049741 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8618005470 con 0x7f86280a37d0 2024-09-13T11:54:59.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f863054b640 5 monclient: authenticate success, global_id 4835 2024-09-13T11:54:59.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f863054b640 10 monclient: _renew_subs 2024-09-13T11:54:59.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f863054b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:54:59.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f863054b640 1 -- 172.21.15.100:0/2617049741 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f862813e060 con 0x7f86280a37d0 2024-09-13T11:54:59.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f861effd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:54:59.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f861effd640 1 -- 172.21.15.100:0/2617049741 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8618012870 con 0x7f86280a37d0 2024-09-13T11:54:59.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f861effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:54:59.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f861effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:54:59.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f861effd640 10 monclient: dump: 2024-09-13T11:54:59.578 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:54:59.578 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.578 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.578 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.578 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:54:59.578 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.578 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:54:59.579 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:54:59.579 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:54:59.579 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f862cae4640 4 set_mon_vals no callback set 2024-09-13T11:54:59.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f863054b640 10 monclient: _renew_subs 2024-09-13T11:54:59.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f863054b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:54:59.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.556+0000 7f863054b640 1 -- 172.21.15.100:0/2617049741 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f86280a5cc0 con 0x7f86280a37d0 2024-09-13T11:54:59.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f863054b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:54:59.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f863054b640 1 --2- 172.21.15.100:0/2617049741 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8628137970 0x7f8628137d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f863054b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f863054b640 1 -- 172.21.15.100:0/2617049741 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f862806f330 con 0x7f8628137970 2024-09-13T11:54:59.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f861effd640 1 -- 172.21.15.100:0/2617049741 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f8618012a10 con 0x7f86280a37d0 2024-09-13T11:54:59.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f862e2e7640 1 --2- 172.21.15.100:0/2617049741 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8628137970 0x7f8628137d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f862e2e7640 10 monclient: get_auth_request con 0x7f8628137970 auth_method 0 2024-09-13T11:54:59.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f862e2e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f862e2e7640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f862e2e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f862e2e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:54:59.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f862e2e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:54:59.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f862e2e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:54:59.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f861effd640 1 --2- 172.21.15.100:0/2617049741 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f860c03dfa0 0x7f860c040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f862e2e7640 10 monclient(hunting): handle_auth_done global_id 4838 payload 824 2024-09-13T11:54:59.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f862e2e7640 1 -- 172.21.15.100:0/2617049741 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f86280a3bb0 con 0x7f8628137970 2024-09-13T11:54:59.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f861effd640 1 -- 172.21.15.100:0/2617049741 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(113..113 src has 1..113) ==== 4738+0+0 (secure 0 0 0) 0x7f86180523b0 con 0x7f86280a37d0 2024-09-13T11:54:59.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f862e2e7640 1 --2- 172.21.15.100:0/2617049741 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8628137970 0x7f8628137d50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f862813e270 tx=0x7f86240019d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:54:59.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f862eae8640 1 --2- 172.21.15.100:0/2617049741 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f860c03dfa0 0x7f860c040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f862eae8640 10 monclient: get_auth_request con 0x7f860c03dfa0 auth_method 0 2024-09-13T11:54:59.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f861effd640 1 -- 172.21.15.100:0/2617049741 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8624019070 con 0x7f8628137970 2024-09-13T11:54:59.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f861effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:54:59.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f861effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:54:59.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f861effd640 10 monclient: dump: 2024-09-13T11:54:59.581 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:54:59.581 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.581 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.581 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.581 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:54:59.581 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.581 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:54:59.582 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:54:59.582 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:54:59.582 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f861effd640 1 -- 172.21.15.100:0/2617049741 <== mon.2 v2:172.21.15.100:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8624005ac0 con 0x7f8628137970 2024-09-13T11:54:59.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f861effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:54:59.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.557+0000 7f861effd640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:54:59.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.558+0000 7f861effd640 1 -- 172.21.15.100:0/2617049741 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8628137970 msgr2=0x7f8628137d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.558+0000 7f861effd640 1 --2- 172.21.15.100:0/2617049741 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8628137970 0x7f8628137d50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f862813e270 tx=0x7f86240019d0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.558+0000 7f861effd640 1 -- 172.21.15.100:0/2617049741 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8628137970 msgr2=0x7f8628137d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:54:59.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.558+0000 7f861effd640 1 --2- 172.21.15.100:0/2617049741 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8628137970 0x7f8628137d50 secure :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f862813e270 tx=0x7f86240019d0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.558+0000 7f862eae8640 1 --2- 172.21.15.100:0/2617049741 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f860c03dfa0 0x7f860c040460 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8620034000 tx=0x7f862002f000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:54:59.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.563+0000 7f863054b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:54:59.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.563+0000 7f863054b640 1 --2- 172.21.15.100:0/2617049741 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8628138290 0x7f8628070f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.563+0000 7f863054b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.563+0000 7f863054b640 1 -- 172.21.15.100:0/2617049741 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f862813f630 con 0x7f8628138290 2024-09-13T11:54:59.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.564+0000 7f862e2e7640 1 --2- 172.21.15.100:0/2617049741 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8628138290 0x7f8628070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.564+0000 7f862e2e7640 10 monclient: get_auth_request con 0x7f8628138290 auth_method 0 2024-09-13T11:54:59.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.564+0000 7f862e2e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.564+0000 7f862e2e7640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.564+0000 7f862e2e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.564+0000 7f862e2e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:54:59.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.564+0000 7f862e2e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:54:59.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.564+0000 7f862e2e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:54:59.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.564+0000 7f862e2e7640 10 monclient(hunting): handle_auth_done global_id 4841 payload 824 2024-09-13T11:54:59.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.564+0000 7f862e2e7640 1 -- 172.21.15.100:0/2617049741 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f862813de30 con 0x7f8628138290 2024-09-13T11:54:59.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.564+0000 7f862e2e7640 1 --2- 172.21.15.100:0/2617049741 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8628138290 0x7f8628070f20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f862813e270 tx=0x7f8624007cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:54:59.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.564+0000 7f861effd640 1 -- 172.21.15.100:0/2617049741 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8624019050 con 0x7f8628138290 2024-09-13T11:54:59.586 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.564+0000 7f861effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:54:59.586 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.564+0000 7f861effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:54:59.586 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.564+0000 7f861effd640 10 monclient: dump: 2024-09-13T11:54:59.586 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:54:59.586 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.586 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.586 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.586 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:54:59.586 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.586 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:54:59.586 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:54:59.586 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:54:59.586 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.587 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.564+0000 7f861effd640 1 -- 172.21.15.100:0/2617049741 <== mon.2 v2:172.21.15.100:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f862401d0d0 con 0x7f8628138290 2024-09-13T11:54:59.587 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.564+0000 7f861effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:54:59.587 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.564+0000 7f861effd640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:54:59.587 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.564+0000 7f861effd640 1 -- 172.21.15.100:0/2617049741 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8628138290 msgr2=0x7f8628070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.587 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.564+0000 7f861effd640 1 --2- 172.21.15.100:0/2617049741 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8628138290 0x7f8628070f20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f862813e270 tx=0x7f8624007cf0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.587 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.564+0000 7f861effd640 1 -- 172.21.15.100:0/2617049741 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8628138290 msgr2=0x7f8628070f20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:54:59.587 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.564+0000 7f861effd640 1 --2- 172.21.15.100:0/2617049741 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8628138290 0x7f8628070f20 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f862813e270 tx=0x7f8624007cf0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.587 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:54:59.587 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:54:59.587 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:54:59.588 INFO:teuthology.orchestra.run.smithi100.stdout: "state": "peon", 2024-09-13T11:54:59.588 INFO:teuthology.orchestra.run.smithi100.stdout: "election_epoch": 16, 2024-09-13T11:54:59.588 INFO:teuthology.orchestra.run.smithi100.stdout: "uptime": 138634, 2024-09-13T11:54:59.588 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum": [ 2024-09-13T11:54:59.588 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-13T11:54:59.588 INFO:teuthology.orchestra.run.smithi100.stdout: 1, 2024-09-13T11:54:59.588 INFO:teuthology.orchestra.run.smithi100.stdout: 2 2024-09-13T11:54:59.588 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:54:59.588 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_age": 0, 2024-09-13T11:54:59.588 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:54:59.588 INFO:teuthology.orchestra.run.smithi100.stdout: "required_con": "2450521708007866372", 2024-09-13T11:54:59.588 INFO:teuthology.orchestra.run.smithi100.stdout: "required_mon": [ 2024-09-13T11:54:59.588 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:54:59.589 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:54:59.589 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:54:59.589 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:54:59.589 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:54:59.589 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:54:59.589 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:54:59.589 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:54:59.589 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:54:59.589 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:54:59.589 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:54:59.589 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:54:59.589 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:54:59.589 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_mon": [ 2024-09-13T11:54:59.590 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:54:59.590 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:54:59.590 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:54:59.590 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:54:59.590 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:54:59.590 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:54:59.590 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:54:59.590 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:54:59.590 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:54:59.590 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:54:59.590 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:54:59.590 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:54:59.590 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.590 INFO:teuthology.orchestra.run.smithi100.stdout: "outside_quorum": [], 2024-09-13T11:54:59.591 INFO:teuthology.orchestra.run.smithi100.stdout: "extra_probe_peers": [], 2024-09-13T11:54:59.591 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_provider": [], 2024-09-13T11:54:59.591 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": { 2024-09-13T11:54:59.591 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-09-13T11:54:59.591 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "0b1f4b4d-fad4-4dd3-86ac-a286d80846c0", 2024-09-13T11:54:59.591 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:54:59.591 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:54:59.591 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release": 19, 2024-09-13T11:54:59.591 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release_name": "squid", 2024-09-13T11:54:59.591 INFO:teuthology.orchestra.run.smithi100.stdout: "election_strategy": 1, 2024-09-13T11:54:59.591 INFO:teuthology.orchestra.run.smithi100.stdout: "disallowed_leaders: ": "", 2024-09-13T11:54:59.591 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false, 2024-09-13T11:54:59.592 INFO:teuthology.orchestra.run.smithi100.stdout: "tiebreaker_mon": "", 2024-09-13T11:54:59.592 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_ranks: ": "", 2024-09-13T11:54:59.592 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:54:59.592 INFO:teuthology.orchestra.run.smithi100.stdout: "persistent": [ 2024-09-13T11:54:59.592 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:54:59.592 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:54:59.592 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:54:59.592 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:54:59.592 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:54:59.592 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:54:59.592 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:54:59.593 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:54:59.593 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:54:59.593 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:54:59.593 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:54:59.593 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:54:59.593 INFO:teuthology.orchestra.run.smithi100.stdout: "optional": [] 2024-09-13T11:54:59.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.593 INFO:teuthology.orchestra.run.smithi100.stdout: "mons": [ 2024-09-13T11:54:59.593 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.593 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:54:59.593 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:54:59.593 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:54:59.593 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:54:59.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.594 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:54:59.594 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3300", 2024-09-13T11:54:59.594 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:54:59.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.594 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:54:59.594 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789", 2024-09-13T11:54:59.594 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:54:59.594 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:54:59.594 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:54:59.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.594 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789/0", 2024-09-13T11:54:59.595 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-13T11:54:59.595 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:54:59.595 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:54:59.595 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:54:59.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.595 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:54:59.595 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:54:59.595 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:54:59.595 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:54:59.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.595 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:54:59.595 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:3300", 2024-09-13T11:54:59.595 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:54:59.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.596 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:54:59.596 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789", 2024-09-13T11:54:59.596 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:54:59.596 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:54:59.596 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:54:59.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.596 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789/0", 2024-09-13T11:54:59.596 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.122:6789/0", 2024-09-13T11:54:59.596 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:54:59.596 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:54:59.596 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:54:59.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.597 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:54:59.597 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:54:59.597 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:54:59.597 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:54:59.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.597 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:54:59.597 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3301", 2024-09-13T11:54:59.597 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:54:59.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.597 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:54:59.597 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790", 2024-09-13T11:54:59.598 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:54:59.598 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:54:59.598 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:54:59.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.598 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790/0", 2024-09-13T11:54:59.598 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-13T11:54:59.598 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:54:59.598 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:54:59.598 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:54:59.598 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:54:59.598 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:54:59.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.598 INFO:teuthology.orchestra.run.smithi100.stdout: "feature_map": { 2024-09-13T11:54:59.599 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": [ 2024-09-13T11:54:59.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.599 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:54:59.599 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:54:59.599 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:54:59.599 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:54:59.599 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:54:59.599 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": [ 2024-09-13T11:54:59.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.599 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:54:59.599 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:54:59.599 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 3 2024-09-13T11:54:59.599 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:54:59.599 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:54:59.600 INFO:teuthology.orchestra.run.smithi100.stdout: "client": [ 2024-09-13T11:54:59.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.600 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:54:59.600 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:54:59.600 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 5 2024-09-13T11:54:59.600 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:54:59.600 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:54:59.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.600 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false 2024-09-13T11:54:59.600 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:54:59.602 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.565+0000 7f863054b640 1 -- 172.21.15.100:0/2617049741 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f860c03dfa0 msgr2=0x7f860c040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.602 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.565+0000 7f863054b640 1 --2- 172.21.15.100:0/2617049741 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f860c03dfa0 0x7f860c040460 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8620034000 tx=0x7f862002f000 comp rx=0 tx=0).stop 2024-09-13T11:54:59.602 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.565+0000 7f863054b640 10 monclient: shutdown 2024-09-13T11:54:59.603 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.565+0000 7f863054b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:54:59.603 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.565+0000 7f863054b640 1 -- 172.21.15.100:0/2617049741 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86280a37d0 msgr2=0x7f862813ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.604 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.565+0000 7f863054b640 1 --2- 172.21.15.100:0/2617049741 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86280a37d0 0x7f862813ae20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f861800c450 tx=0x7f861800c910 comp rx=0 tx=0).stop 2024-09-13T11:54:59.604 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.565+0000 7f862eae8640 1 -- 172.21.15.100:0/2617049741 reap_dead start 2024-09-13T11:54:59.604 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.565+0000 7f862eae8640 1 -- 172.21.15.100:0/2617049741 reap_dead start 2024-09-13T11:54:59.604 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.565+0000 7f863054b640 1 -- 172.21.15.100:0/2617049741 shutdown_connections 2024-09-13T11:54:59.604 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.565+0000 7f863054b640 1 -- 172.21.15.100:0/2617049741 >> 172.21.15.100:0/2617049741 conn(0x7f86280a83f0 msgr2=0x7f862809da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:54:59.604 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.565+0000 7f863054b640 1 -- 172.21.15.100:0/2617049741 shutdown_connections 2024-09-13T11:54:59.604 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.565+0000 7f863054b640 1 -- 172.21.15.100:0/2617049741 wait complete. 2024-09-13T11:54:59.604 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T11:54:59.605 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-13T11:54:59.605 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-13T11:54:59.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.702+0000 7f5e73e31640 10 monclient: get_monmap_and_config 2024-09-13T11:54:59.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.702+0000 7f5e73e31640 10 monclient: build_initial_monmap 2024-09-13T11:54:59.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.702+0000 7f5e73e31640 10 monclient: monmap: 2024-09-13T11:54:59.704 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:54:59.704 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.704 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:54:59.703101+0000 2024-09-13T11:54:59.704 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:54:59.703101+0000 2024-09-13T11:54:59.704 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:54:59.704 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.704 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:54:59.704 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:54:59.704 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:54:59.705 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:54:59.705 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:54:59.705 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.702+0000 7f5e73e31640 1 Processor -- start 2024-09-13T11:54:59.714 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e73e31640 1 -- start start 2024-09-13T11:54:59.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e73e31640 10 monclient: init 2024-09-13T11:54:59.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e73e31640 10 monclient: _reopen_session rank -1 2024-09-13T11:54:59.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e73e31640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-13T11:54:59.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e73e31640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e6c06ea50 0x7f5e6c0a2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e73e31640 10 monclient(hunting): picked mon.noname-b con 0x7f5e6c06ea50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:54:59.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e73e31640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e6c06ee30 addr v1:172.21.15.122:6789/0 2024-09-13T11:54:59.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e73e31640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5e6c13c9f0 0x7f5e6c13ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e73e31640 10 monclient(hunting): picked mon.noname-c con 0x7f5e6c13c9f0 addr v2:172.21.15.122:3300/0 2024-09-13T11:54:59.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e73e31640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e6c0a1590 con 0x7f5e6c06ee30 2024-09-13T11:54:59.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e73e31640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e73e31640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f5e6c0a1870 con 0x7f5e6c13c9f0 2024-09-13T11:54:59.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e73e31640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.715 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e73e31640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f5e6c0a1b50 con 0x7f5e6c06ea50 2024-09-13T11:54:59.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e73e31640 10 monclient(hunting): _renew_subs 2024-09-13T11:54:59.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e73e31640 10 monclient(hunting): authenticate will time out at 880.296603s 2024-09-13T11:54:59.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e71bcd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e6c06ea50 0x7f5e6c0a2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e71bcd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e6c06ea50 0x7f5e6c0a2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:36890/0 (socket says 172.21.15.100:36890) 2024-09-13T11:54:59.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e71bcd640 1 -- 172.21.15.100:0/305273173 learned_addr learned my addr 172.21.15.100:0/305273173 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:54:59.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e71bcd640 10 monclient(hunting): get_auth_request con 0x7f5e6c06ea50 auth_method 0 2024-09-13T11:54:59.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e71bcd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e71bcd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e71bcd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e723ce640 1 --2- 172.21.15.100:0/305273173 >> v2:172.21.15.122:3300/0 conn(0x7f5e6c13c9f0 0x7f5e6c13ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e723ce640 10 monclient(hunting): get_auth_request con 0x7f5e6c13c9f0 auth_method 0 2024-09-13T11:54:59.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e723ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e723ce640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e723ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e71bcd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:54:59.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e71bcd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:54:59.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e71bcd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:54:59.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e723ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:54:59.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e723ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:54:59.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e723ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:54:59.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e71bcd640 10 monclient(hunting): handle_auth_done global_id 4844 payload 274 2024-09-13T11:54:59.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e71bcd640 1 -- 172.21.15.100:0/305273173 >> v2:172.21.15.122:3300/0 conn(0x7f5e6c13c9f0 msgr2=0x7f5e6c13ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:54:59.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e71bcd640 1 --2- 172.21.15.100:0/305273173 >> v2:172.21.15.122:3300/0 conn(0x7f5e6c13c9f0 0x7f5e6c13ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.703+0000 7f5e71bcd640 1 -- 172.21.15.100:0/305273173 >> v1:172.21.15.122:6789/0 conn(0x7f5e6c06ee30 legacy=0x7f5e6c06f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.704+0000 7f5e71bcd640 10 monclient: _finish_hunting 0 2024-09-13T11:54:59.717 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.704+0000 7f5e71bcd640 1 monclient: found mon.noname-b 2024-09-13T11:54:59.718 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.704+0000 7f5e71bcd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:54:59.718 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.704+0000 7f5e71bcd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:54:59.718 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.704+0000 7f5e71bcd640 1 -- 172.21.15.100:0/305273173 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e6c06fca0 con 0x7f5e6c06ea50 2024-09-13T11:54:59.718 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.704+0000 7f5e71bcd640 1 --2- 172.21.15.100:0/305273173 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e6c06ea50 0x7f5e6c0a2860 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5e60009870 tx=0x7f5e6002a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f792aebf9fd651f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:54:59.718 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.704+0000 7f5e70bcb640 1 -- 172.21.15.100:0/305273173 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e60037070 con 0x7f5e6c06ea50 2024-09-13T11:54:59.718 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.704+0000 7f5e70bcb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:54:59.718 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.704+0000 7f5e70bcb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:54:59.718 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.704+0000 7f5e70bcb640 10 monclient: dump: 2024-09-13T11:54:59.718 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:54:59.718 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.718 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.718 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.718 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:54:59.719 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.719 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:54:59.719 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:54:59.719 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:54:59.719 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.704+0000 7f5e70bcb640 10 monclient: _finish_auth 0 2024-09-13T11:54:59.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.704+0000 7f5e70bcb640 10 monclient: _check_auth_tickets 2024-09-13T11:54:59.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.704+0000 7f5e70bcb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:54:59.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.704+0000 7f5e73e31640 5 monclient: authenticate success, global_id 4844 2024-09-13T11:54:59.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.704+0000 7f5e70bcb640 1 -- 172.21.15.100:0/305273173 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e60032440 con 0x7f5e6c06ea50 2024-09-13T11:54:59.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.704+0000 7f5e73e31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:54:59.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.704+0000 7f5e70bcb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:54:59.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.704+0000 7f5e73e31640 10 monclient: get_monmap_and_config success 2024-09-13T11:54:59.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.704+0000 7f5e73e31640 4 set_mon_vals no callback set 2024-09-13T11:54:59.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.704+0000 7f5e70bcb640 1 -- 172.21.15.100:0/305273173 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e60032810 con 0x7f5e6c06ea50 2024-09-13T11:54:59.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.704+0000 7f5e70bcb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:54:59.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.704+0000 7f5e70bcb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:54:59.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.704+0000 7f5e70bcb640 10 monclient: dump: 2024-09-13T11:54:59.720 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:54:59.720 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.720 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.720 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.721 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:54:59.721 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.721 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:54:59.721 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:54:59.721 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:54:59.721 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.705+0000 7f5e73e31640 10 monclient: shutdown 2024-09-13T11:54:59.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.705+0000 7f5e73e31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:54:59.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.705+0000 7f5e73e31640 1 -- 172.21.15.100:0/305273173 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e6c06ea50 msgr2=0x7f5e6c0a2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.705+0000 7f5e73e31640 1 --2- 172.21.15.100:0/305273173 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e6c06ea50 0x7f5e6c0a2860 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5e60009870 tx=0x7f5e6002a130 comp rx=0 tx=0).stop 2024-09-13T11:54:59.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.705+0000 7f5e73e31640 1 -- 172.21.15.100:0/305273173 shutdown_connections 2024-09-13T11:54:59.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.705+0000 7f5e73e31640 1 --2- 172.21.15.100:0/305273173 >> v2:172.21.15.122:3300/0 conn(0x7f5e6c13c9f0 0x7f5e6c13ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.705+0000 7f5e73e31640 1 --2- 172.21.15.100:0/305273173 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e6c06ea50 0x7f5e6c0a2860 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.705+0000 7f5e73e31640 1 -- 172.21.15.100:0/305273173 >> 172.21.15.100:0/305273173 conn(0x7f5e6c0a68e0 msgr2=0x7f5e6c0a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:54:59.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.705+0000 7f5e73e31640 1 -- 172.21.15.100:0/305273173 shutdown_connections 2024-09-13T11:54:59.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.705+0000 7f5e73e31640 1 -- 172.21.15.100:0/305273173 wait complete. 2024-09-13T11:54:59.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e73e31640 10 monclient: build_initial_monmap 2024-09-13T11:54:59.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e73e31640 10 monclient: monmap: 2024-09-13T11:54:59.722 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:54:59.722 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:54:59.722 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:54:59.722 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:54:59.722 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:54:59.722 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.722 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:54:59.722 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:54:59.723 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:54:59.723 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e73e31640 1 Processor -- start 2024-09-13T11:54:59.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e73e31640 1 -- start start 2024-09-13T11:54:59.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e73e31640 10 monclient: init 2024-09-13T11:54:59.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e73e31640 10 monclient: _reopen_session rank -1 2024-09-13T11:54:59.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e73e31640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:54:59.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e73e31640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e6c06ea50 0x7f5e6c137320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e73e31640 10 monclient(hunting): picked mon.noname-a con 0x7f5e6c06ea50 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:54:59.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e73e31640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e6c06ee30 0x7f5e6c137860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e73e31640 10 monclient(hunting): picked mon.noname-c con 0x7f5e6c06ee30 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:54:59.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e73e31640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c13c9f0 0x7f5e6c09ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e73e31640 10 monclient(hunting): picked mon.noname-b con 0x7f5e6c13c9f0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:54:59.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e73e31640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e73e31640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f5e6c0a5be0 con 0x7f5e6c06ea50 2024-09-13T11:54:59.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e73e31640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e73e31640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f5e6c0a5a60 con 0x7f5e6c13c9f0 2024-09-13T11:54:59.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e73e31640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e73e31640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f5e6c13f320 con 0x7f5e6c06ee30 2024-09-13T11:54:59.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e73e31640 10 monclient(hunting): _renew_subs 2024-09-13T11:54:59.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e73e31640 10 monclient(hunting): authenticate will time out at 880.299826s 2024-09-13T11:54:59.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e71bcd640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e6c06ea50 0x7f5e6c137320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e713cc640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e6c06ee30 0x7f5e6c137860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e71bcd640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e6c06ea50 0x7f5e6c137320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53428/0 (socket says 172.21.15.100:53428) 2024-09-13T11:54:59.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e713cc640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e6c06ee30 0x7f5e6c137860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:36894/0 (socket says 172.21.15.100:36894) 2024-09-13T11:54:59.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e71bcd640 1 -- 172.21.15.100:0/996729868 learned_addr learned my addr 172.21.15.100:0/996729868 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:54:59.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e71bcd640 10 monclient(hunting): get_auth_request con 0x7f5e6c06ea50 auth_method 0 2024-09-13T11:54:59.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e71bcd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e71bcd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e71bcd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e713cc640 10 monclient(hunting): get_auth_request con 0x7f5e6c06ee30 auth_method 0 2024-09-13T11:54:59.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e713cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e713cc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.706+0000 7f5e713cc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e71bcd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:54:59.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e71bcd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:54:59.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e71bcd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:54:59.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e713cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:54:59.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e713cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:54:59.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e723ce640 1 --2- 172.21.15.100:0/996729868 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c13c9f0 0x7f5e6c09ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e713cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:54:59.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e723ce640 10 monclient(hunting): get_auth_request con 0x7f5e6c13c9f0 auth_method 0 2024-09-13T11:54:59.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e723ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e723ce640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e723ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e723ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:54:59.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e723ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:54:59.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e723ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:54:59.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e713cc640 10 monclient(hunting): handle_auth_done global_id 4847 payload 824 2024-09-13T11:54:59.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e713cc640 1 -- 172.21.15.100:0/996729868 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c13c9f0 msgr2=0x7f5e6c09ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.728 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e713cc640 1 --2- 172.21.15.100:0/996729868 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c13c9f0 0x7f5e6c09ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.728 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e713cc640 1 -- 172.21.15.100:0/996729868 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e6c06ea50 msgr2=0x7f5e6c137320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.728 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e713cc640 1 --2- 172.21.15.100:0/996729868 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e6c06ea50 0x7f5e6c137320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.728 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e713cc640 10 monclient: _finish_hunting 0 2024-09-13T11:54:59.728 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e713cc640 1 monclient: found mon.noname-c 2024-09-13T11:54:59.729 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e713cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:54:59.729 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e713cc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:54:59.729 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e713cc640 1 -- 172.21.15.100:0/996729868 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e6c121c10 con 0x7f5e6c06ee30 2024-09-13T11:54:59.729 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e71bcd640 1 --2- 172.21.15.100:0/996729868 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e6c06ea50 0x7f5e6c137320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:54:59.729 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e713cc640 1 --2- 172.21.15.100:0/996729868 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e6c06ee30 0x7f5e6c137860 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5e6800c3c0 tx=0x7f5e6800c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:54:59.729 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e5e7fc640 1 -- 172.21.15.100:0/996729868 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e68017070 con 0x7f5e6c06ee30 2024-09-13T11:54:59.729 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:54:59.729 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e5e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:54:59.730 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e5e7fc640 10 monclient: dump: 2024-09-13T11:54:59.730 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:54:59.730 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.730 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.730 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.730 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:54:59.730 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.730 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:54:59.730 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:54:59.730 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:54:59.730 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.730 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e5e7fc640 10 monclient: _finish_auth 0 2024-09-13T11:54:59.730 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e5e7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:54:59.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:54:59.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e73e31640 5 monclient: authenticate success, global_id 4847 2024-09-13T11:54:59.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e5e7fc640 1 -- 172.21.15.100:0/996729868 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e680052d0 con 0x7f5e6c06ee30 2024-09-13T11:54:59.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e5e7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:54:59.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e5e7fc640 1 -- 172.21.15.100:0/996729868 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e68003830 con 0x7f5e6c06ee30 2024-09-13T11:54:59.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e73e31640 10 monclient: _renew_subs 2024-09-13T11:54:59.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e73e31640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:54:59.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e73e31640 1 -- 172.21.15.100:0/996729868 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e6c121ef0 con 0x7f5e6c06ee30 2024-09-13T11:54:59.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e5ffff640 4 set_mon_vals no callback set 2024-09-13T11:54:59.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:54:59.732 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e5e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:54:59.732 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e5e7fc640 10 monclient: dump: 2024-09-13T11:54:59.732 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:54:59.732 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.732 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.732 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.732 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:54:59.732 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.732 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:54:59.733 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:54:59.733 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:54:59.733 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e73e31640 10 monclient: _renew_subs 2024-09-13T11:54:59.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e73e31640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:54:59.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.707+0000 7f5e73e31640 1 -- 172.21.15.100:0/996729868 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e6c121930 con 0x7f5e6c06ee30 2024-09-13T11:54:59.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.708+0000 7f5e73e31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:54:59.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.708+0000 7f5e73e31640 1 --2- 172.21.15.100:0/996729868 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c09f8d0 0x7f5e6c02d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.708+0000 7f5e73e31640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.708+0000 7f5e73e31640 1 -- 172.21.15.100:0/996729868 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f5e6c0a2860 con 0x7f5e6c09f8d0 2024-09-13T11:54:59.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.708+0000 7f5e5e7fc640 1 -- 172.21.15.100:0/996729868 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f5e6801a070 con 0x7f5e6c06ee30 2024-09-13T11:54:59.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.708+0000 7f5e71bcd640 1 --2- 172.21.15.100:0/996729868 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c09f8d0 0x7f5e6c02d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.708+0000 7f5e71bcd640 10 monclient: get_auth_request con 0x7f5e6c09f8d0 auth_method 0 2024-09-13T11:54:59.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.708+0000 7f5e71bcd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.708+0000 7f5e71bcd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.708+0000 7f5e71bcd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.708+0000 7f5e71bcd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:54:59.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.708+0000 7f5e71bcd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:54:59.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.708+0000 7f5e71bcd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:54:59.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.708+0000 7f5e5e7fc640 1 --2- 172.21.15.100:0/996729868 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f5e5003dfd0 0x7f5e50040490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.708+0000 7f5e71bcd640 10 monclient(hunting): handle_auth_done global_id 4951 payload 824 2024-09-13T11:54:59.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.708+0000 7f5e5e7fc640 1 -- 172.21.15.100:0/996729868 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(113..113 src has 1..113) ==== 4738+0+0 (secure 0 0 0) 0x7f5e680519e0 con 0x7f5e6c06ee30 2024-09-13T11:54:59.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.708+0000 7f5e71bcd640 1 -- 172.21.15.100:0/996729868 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5e6c13c0f0 con 0x7f5e6c09f8d0 2024-09-13T11:54:59.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.708+0000 7f5e723ce640 1 --2- 172.21.15.100:0/996729868 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f5e5003dfd0 0x7f5e50040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.708+0000 7f5e723ce640 10 monclient: get_auth_request con 0x7f5e5003dfd0 auth_method 0 2024-09-13T11:54:59.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.708+0000 7f5e71bcd640 1 --2- 172.21.15.100:0/996729868 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c09f8d0 0x7f5e6c02d1f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5e60035040 tx=0x7f5e600046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:54:59.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.709+0000 7f5e5e7fc640 1 -- 172.21.15.100:0/996729868 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e60043070 con 0x7f5e6c09f8d0 2024-09-13T11:54:59.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.709+0000 7f5e5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:54:59.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.709+0000 7f5e5e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:54:59.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.709+0000 7f5e5e7fc640 10 monclient: dump: 2024-09-13T11:54:59.735 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:54:59.735 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.735 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.735 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.735 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:54:59.735 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.735 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:54:59.735 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:54:59.735 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:54:59.735 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.709+0000 7f5e723ce640 1 --2- 172.21.15.100:0/996729868 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f5e5003dfd0 0x7f5e50040490 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c138540 tx=0x7f5e6400b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:54:59.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.709+0000 7f5e5e7fc640 1 -- 172.21.15.100:0/996729868 <== mon.1 v2:172.21.15.122:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5e60032440 con 0x7f5e6c09f8d0 2024-09-13T11:54:59.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.709+0000 7f5e5e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:54:59.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.709+0000 7f5e5e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:54:59.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.709+0000 7f5e5e7fc640 1 -- 172.21.15.100:0/996729868 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c09f8d0 msgr2=0x7f5e6c02d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.709+0000 7f5e5e7fc640 1 --2- 172.21.15.100:0/996729868 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c09f8d0 0x7f5e6c02d1f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5e60035040 tx=0x7f5e600046c0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.709+0000 7f5e5e7fc640 1 -- 172.21.15.100:0/996729868 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c09f8d0 msgr2=0x7f5e6c02d1f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:54:59.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.709+0000 7f5e5e7fc640 1 --2- 172.21.15.100:0/996729868 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c09f8d0 0x7f5e6c02d1f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.715+0000 7f5e73e31640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:54:59.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.715+0000 7f5e73e31640 1 --2- 172.21.15.100:0/996729868 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c09fcb0 0x7f5e6c02d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.715+0000 7f5e73e31640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.715+0000 7f5e73e31640 1 -- 172.21.15.100:0/996729868 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f5e6c069100 con 0x7f5e6c09fcb0 2024-09-13T11:54:59.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.715+0000 7f5e71bcd640 1 --2- 172.21.15.100:0/996729868 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c09fcb0 0x7f5e6c02d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.715+0000 7f5e71bcd640 10 monclient: get_auth_request con 0x7f5e6c09fcb0 auth_method 0 2024-09-13T11:54:59.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.715+0000 7f5e71bcd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.715+0000 7f5e71bcd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.715+0000 7f5e71bcd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.715+0000 7f5e71bcd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:54:59.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.715+0000 7f5e71bcd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:54:59.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.715+0000 7f5e71bcd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:54:59.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.716+0000 7f5e71bcd640 10 monclient(hunting): handle_auth_done global_id 4954 payload 824 2024-09-13T11:54:59.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.716+0000 7f5e71bcd640 1 -- 172.21.15.100:0/996729868 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5e6c1206a0 con 0x7f5e6c09fcb0 2024-09-13T11:54:59.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.716+0000 7f5e71bcd640 1 --2- 172.21.15.100:0/996729868 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c09fcb0 0x7f5e6c02d730 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c0a29e0 tx=0x7f5e60009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:54:59.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.716+0000 7f5e5e7fc640 1 -- 172.21.15.100:0/996729868 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e60037040 con 0x7f5e6c09fcb0 2024-09-13T11:54:59.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.716+0000 7f5e5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:54:59.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.716+0000 7f5e5e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:54:59.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.716+0000 7f5e5e7fc640 10 monclient: dump: 2024-09-13T11:54:59.740 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:54:59.740 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.740 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.740 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:54:59.740 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:54:59.740 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.740 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:54:59.740 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:54:59.741 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:54:59.741 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.716+0000 7f5e5e7fc640 1 -- 172.21.15.100:0/996729868 <== mon.1 v2:172.21.15.122:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f5e60003d80 con 0x7f5e6c09fcb0 2024-09-13T11:54:59.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.716+0000 7f5e5e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:54:59.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.716+0000 7f5e5e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:54:59.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.716+0000 7f5e5e7fc640 1 -- 172.21.15.100:0/996729868 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c09fcb0 msgr2=0x7f5e6c02d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.716+0000 7f5e5e7fc640 1 --2- 172.21.15.100:0/996729868 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c09fcb0 0x7f5e6c02d730 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c0a29e0 tx=0x7f5e60009240 comp rx=0 tx=0).stop 2024-09-13T11:54:59.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.716+0000 7f5e5e7fc640 1 -- 172.21.15.100:0/996729868 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c09fcb0 msgr2=0x7f5e6c02d730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:54:59.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.716+0000 7f5e5e7fc640 1 --2- 172.21.15.100:0/996729868 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c09fcb0 0x7f5e6c02d730 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.741 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:54:59.742 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:54:59.742 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:54:59.742 INFO:teuthology.orchestra.run.smithi100.stdout: "state": "peon", 2024-09-13T11:54:59.742 INFO:teuthology.orchestra.run.smithi100.stdout: "election_epoch": 16, 2024-09-13T11:54:59.742 INFO:teuthology.orchestra.run.smithi100.stdout: "uptime": 138746, 2024-09-13T11:54:59.742 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum": [ 2024-09-13T11:54:59.742 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-13T11:54:59.742 INFO:teuthology.orchestra.run.smithi100.stdout: 1, 2024-09-13T11:54:59.742 INFO:teuthology.orchestra.run.smithi100.stdout: 2 2024-09-13T11:54:59.742 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:54:59.742 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_age": 0, 2024-09-13T11:54:59.742 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:54:59.742 INFO:teuthology.orchestra.run.smithi100.stdout: "required_con": "2450521708007866372", 2024-09-13T11:54:59.743 INFO:teuthology.orchestra.run.smithi100.stdout: "required_mon": [ 2024-09-13T11:54:59.743 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:54:59.743 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:54:59.743 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:54:59.743 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:54:59.743 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:54:59.743 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:54:59.743 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:54:59.743 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:54:59.743 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:54:59.743 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:54:59.743 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:54:59.743 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:54:59.744 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:54:59.744 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_mon": [ 2024-09-13T11:54:59.744 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:54:59.744 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:54:59.744 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:54:59.744 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:54:59.744 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:54:59.744 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:54:59.744 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:54:59.744 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:54:59.744 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:54:59.744 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:54:59.744 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:54:59.745 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:54:59.745 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.745 INFO:teuthology.orchestra.run.smithi100.stdout: "outside_quorum": [], 2024-09-13T11:54:59.745 INFO:teuthology.orchestra.run.smithi100.stdout: "extra_probe_peers": [], 2024-09-13T11:54:59.745 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_provider": [], 2024-09-13T11:54:59.745 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": { 2024-09-13T11:54:59.745 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-09-13T11:54:59.745 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "0b1f4b4d-fad4-4dd3-86ac-a286d80846c0", 2024-09-13T11:54:59.745 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:54:59.745 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:54:59.745 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release": 19, 2024-09-13T11:54:59.745 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release_name": "squid", 2024-09-13T11:54:59.745 INFO:teuthology.orchestra.run.smithi100.stdout: "election_strategy": 1, 2024-09-13T11:54:59.745 INFO:teuthology.orchestra.run.smithi100.stdout: "disallowed_leaders: ": "", 2024-09-13T11:54:59.746 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false, 2024-09-13T11:54:59.746 INFO:teuthology.orchestra.run.smithi100.stdout: "tiebreaker_mon": "", 2024-09-13T11:54:59.746 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_ranks: ": "", 2024-09-13T11:54:59.746 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:54:59.746 INFO:teuthology.orchestra.run.smithi100.stdout: "persistent": [ 2024-09-13T11:54:59.746 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:54:59.746 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:54:59.746 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:54:59.746 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:54:59.746 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:54:59.746 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:54:59.746 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:54:59.746 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:54:59.747 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:54:59.747 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:54:59.747 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:54:59.747 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:54:59.747 INFO:teuthology.orchestra.run.smithi100.stdout: "optional": [] 2024-09-13T11:54:59.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.747 INFO:teuthology.orchestra.run.smithi100.stdout: "mons": [ 2024-09-13T11:54:59.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.747 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:54:59.747 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:54:59.747 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:54:59.747 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:54:59.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.748 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:54:59.748 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3300", 2024-09-13T11:54:59.748 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:54:59.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.748 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:54:59.748 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789", 2024-09-13T11:54:59.748 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:54:59.748 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:54:59.748 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:54:59.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.748 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789/0", 2024-09-13T11:54:59.748 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-13T11:54:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:54:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:54:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:54:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:54:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:54:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:54:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:54:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:54:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:3300", 2024-09-13T11:54:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:54:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:54:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789", 2024-09-13T11:54:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:54:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:54:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:54:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789/0", 2024-09-13T11:54:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.122:6789/0", 2024-09-13T11:54:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:54:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:54:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:54:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:54:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:54:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:54:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:54:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:54:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3301", 2024-09-13T11:54:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:54:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:54:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790", 2024-09-13T11:54:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:54:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:54:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:54:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790/0", 2024-09-13T11:54:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-13T11:54:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:54:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:54:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:54:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:54:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:54:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: "feature_map": { 2024-09-13T11:54:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": [ 2024-09-13T11:54:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:54:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:54:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:54:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:54:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:54:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": [ 2024-09-13T11:54:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:54:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:54:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 3 2024-09-13T11:54:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:54:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:54:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: "client": [ 2024-09-13T11:54:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:54:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:54:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 5 2024-09-13T11:54:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:54:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:54:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": [ 2024-09-13T11:54:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:54:59.755 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:54:59.755 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:54:59.755 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:54:59.755 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:54:59.755 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:54:59.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:54:59.755 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false 2024-09-13T11:54:59.755 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:54:59.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.717+0000 7f5e73e31640 1 -- 172.21.15.100:0/996729868 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f5e5003dfd0 msgr2=0x7f5e50040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.717+0000 7f5e73e31640 1 --2- 172.21.15.100:0/996729868 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f5e5003dfd0 0x7f5e50040490 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c138540 tx=0x7f5e6400b040 comp rx=0 tx=0).stop 2024-09-13T11:54:59.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.717+0000 7f5e73e31640 10 monclient: shutdown 2024-09-13T11:54:59.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.717+0000 7f5e73e31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:54:59.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.717+0000 7f5e73e31640 1 -- 172.21.15.100:0/996729868 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e6c06ee30 msgr2=0x7f5e6c137860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.717+0000 7f5e73e31640 1 --2- 172.21.15.100:0/996729868 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5e6c06ee30 0x7f5e6c137860 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5e6800c3c0 tx=0x7f5e6800c880 comp rx=0 tx=0).stop 2024-09-13T11:54:59.758 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.717+0000 7f5e723ce640 1 -- 172.21.15.100:0/996729868 reap_dead start 2024-09-13T11:54:59.758 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.717+0000 7f5e723ce640 1 -- 172.21.15.100:0/996729868 reap_dead start 2024-09-13T11:54:59.758 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.717+0000 7f5e73e31640 1 -- 172.21.15.100:0/996729868 shutdown_connections 2024-09-13T11:54:59.758 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.717+0000 7f5e73e31640 1 -- 172.21.15.100:0/996729868 >> 172.21.15.100:0/996729868 conn(0x7f5e6c0a68e0 msgr2=0x7f5e6c13d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:54:59.758 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.717+0000 7f5e73e31640 1 -- 172.21.15.100:0/996729868 shutdown_connections 2024-09-13T11:54:59.758 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.717+0000 7f5e73e31640 1 -- 172.21.15.100:0/996729868 wait complete. 2024-09-13T11:54:59.758 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T11:54:59.758 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-13T11:54:59.758 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-13T11:54:59.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.855+0000 7ff341975640 10 monclient: get_monmap_and_config 2024-09-13T11:54:59.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.855+0000 7ff341975640 10 monclient: build_initial_monmap 2024-09-13T11:54:59.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.855+0000 7ff341975640 10 monclient: monmap: 2024-09-13T11:54:59.857 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:54:59.857 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:54:59.857 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:54:59.856128+0000 2024-09-13T11:54:59.857 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:54:59.856128+0000 2024-09-13T11:54:59.857 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:54:59.857 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:54:59.857 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:54:59.857 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:54:59.857 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:54:59.857 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:54:59.857 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:54:59.857 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:54:59.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.855+0000 7ff341975640 1 Processor -- start 2024-09-13T11:54:59.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.855+0000 7ff341975640 1 -- start start 2024-09-13T11:54:59.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.855+0000 7ff341975640 10 monclient: init 2024-09-13T11:54:59.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff341975640 10 monclient: _reopen_session rank -1 2024-09-13T11:54:59.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff341975640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-13T11:54:59.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff341975640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff33c06ea50 addr v1:172.21.15.100:6789/0 2024-09-13T11:54:59.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff341975640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff33c0a2860 0x7ff33c06f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff341975640 10 monclient(hunting): picked mon.noname-a con 0x7ff33c0a2860 addr v2:172.21.15.100:3300/0 2024-09-13T11:54:59.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff341975640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff33c13c9f0 0x7ff33c13ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:54:59.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff341975640 10 monclient(hunting): picked mon.noname-b con 0x7ff33c13c9f0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:54:59.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff341975640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff33c06fa80 con 0x7ff33c06ea50 2024-09-13T11:54:59.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff341975640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff341975640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7ff33c0a2cb0 con 0x7ff33c0a2860 2024-09-13T11:54:59.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff341975640 10 monclient(hunting): start opening mon connection 2024-09-13T11:54:59.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff341975640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7ff33c0a15e0 con 0x7ff33c13c9f0 2024-09-13T11:54:59.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff341975640 10 monclient(hunting): _renew_subs 2024-09-13T11:54:59.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff341975640 10 monclient(hunting): authenticate will time out at 880.449548s 2024-09-13T11:54:59.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33a7fc640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff33c0a2860 0x7ff33c06f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33affd640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7ff33c06ea50 0x7ff33c06ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:48244/0 (socket says 172.21.15.100:48244) 2024-09-13T11:54:59.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33affd640 1 -- 172.21.15.100:0/2096276329 learned_addr learned my addr 172.21.15.100:0/2096276329 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:54:59.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33a7fc640 10 monclient(hunting): get_auth_request con 0x7ff33c0a2860 auth_method 0 2024-09-13T11:54:59.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33b7fe640 1 --2- 172.21.15.100:0/2096276329 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff33c13c9f0 0x7ff33c13ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:54:59.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33b7fe640 10 monclient(hunting): get_auth_request con 0x7ff33c13c9f0 auth_method 0 2024-09-13T11:54:59.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:54:59.862 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:54:59.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:54:59.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:54:59.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:54:59.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:54:59.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:54:59.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:54:59.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:54:59.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33a7fc640 10 monclient(hunting): handle_auth_done global_id 14124 payload 274 2024-09-13T11:54:59.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33a7fc640 1 -- 172.21.15.100:0/2096276329 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff33c13c9f0 msgr2=0x7ff33c13ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:54:59.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33a7fc640 1 --2- 172.21.15.100:0/2096276329 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff33c13c9f0 0x7ff33c13ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:54:59.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33a7fc640 1 -- 172.21.15.100:0/2096276329 >> v1:172.21.15.100:6789/0 conn(0x7ff33c06ea50 legacy=0x7ff33c06ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:54:59.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33a7fc640 10 monclient: _finish_hunting 0 2024-09-13T11:54:59.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33a7fc640 1 monclient: found mon.noname-a 2024-09-13T11:54:59.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:54:59.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:54:59.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff33a7fc640 1 -- 172.21.15.100:0/2096276329 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff33c0a1870 con 0x7ff33c0a2860 2024-09-13T11:54:59.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff339ffb640 1 -- 172.21.15.100:0/2096276329 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 355104955 0 0) 0x7ff334001490 con 0x7ff33c06ea50 2024-09-13T11:54:59.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.856+0000 7ff339ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:54:59.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.857+0000 7ff33a7fc640 1 -- 172.21.15.100:0/2096276329 >> v2:172.21.15.100:3300/0 conn(0x7ff33c0a2860 msgr2=0x7ff33c06f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T11:54:59.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.857+0000 7ff33a7fc640 1 -- 172.21.15.100:0/2096276329 >> v2:172.21.15.100:3300/0 conn(0x7ff33c0a2860 msgr2=0x7ff33c06f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:54:59.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.857+0000 7ff33a7fc640 1 --2- 172.21.15.100:0/2096276329 >> v2:172.21.15.100:3300/0 conn(0x7ff33c0a2860 0x7ff33c06f540 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7ff3240097d0 tx=0x7ff32402a0a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:54:59.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:54:59.857+0000 7ff33a7fc640 1 --2- 172.21.15.100:0/2096276329 >> v2:172.21.15.100:3300/0 conn(0x7ff33c0a2860 0x7ff33c06f540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:55:00.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff33a7fc640 1 --2- 172.21.15.100:0/2096276329 >> v2:172.21.15.100:3300/0 conn(0x7ff33c0a2860 0x7ff33c06f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:00.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff33a7fc640 10 monclient: get_auth_request con 0x7ff33c0a2860 auth_method 0 2024-09-13T11:55:00.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff33a7fc640 0 --2- 172.21.15.100:0/2096276329 >> v2:172.21.15.100:3300/0 conn(0x7ff33c0a2860 0x7ff33c06f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-13T11:55:00.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff33a7fc640 1 --2- 172.21.15.100:0/2096276329 >> v2:172.21.15.100:3300/0 conn(0x7ff33c0a2860 0x7ff33c06f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:00.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff339ffb640 10 monclient: ms_handle_reset current mon v2:172.21.15.100:3300/0 2024-09-13T11:55:00.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff339ffb640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:00.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff339ffb640 1 -- 172.21.15.100:0/2096276329 >> v2:172.21.15.100:3300/0 conn(0x7ff33c0a2860 msgr2=0x7ff33c06f540 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-13T11:55:00.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff339ffb640 1 --2- 172.21.15.100:0/2096276329 >> v2:172.21.15.100:3300/0 conn(0x7ff33c0a2860 0x7ff33c06f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:00.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff339ffb640 10 monclient: _add_conns ranks=[1,4] 2024-09-13T11:55:00.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff339ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff3280034c0 addr v1:172.21.15.122:6789/0 2024-09-13T11:55:00.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff339ffb640 1 --2- 172.21.15.100:0/2096276329 >> v2:172.21.15.122:3300/0 conn(0x7ff3280073c0 0x7ff328009880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:00.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff339ffb640 10 monclient(hunting): picked mon.noname-c con 0x7ff3280073c0 addr v2:172.21.15.122:3300/0 2024-09-13T11:55:00.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff339ffb640 1 -- 172.21.15.100:0/2096276329 --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff334001490 con 0x7ff3280034c0 2024-09-13T11:55:00.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff339ffb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:00.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff339ffb640 1 -- 172.21.15.100:0/2096276329 --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7ff32800b180 con 0x7ff3280073c0 2024-09-13T11:55:00.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff339ffb640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:00.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff339ffb640 10 monclient(hunting): ms_handle_reset stray mon v2:172.21.15.100:3300/0 2024-09-13T11:55:00.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff33a7fc640 1 --2- 172.21.15.100:0/2096276329 >> v2:172.21.15.122:3300/0 conn(0x7ff3280073c0 0x7ff328009880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:00.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff33a7fc640 10 monclient(hunting): get_auth_request con 0x7ff3280073c0 auth_method 0 2024-09-13T11:55:00.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff33a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:00.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff33a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:00.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff33a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:55:00.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff33a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:00.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff33a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:00.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff33a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:55:00.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff339ffb640 1 -- 172.21.15.100:0/2096276329 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 171101625 0 0) 0x7ff334001490 con 0x7ff3280034c0 2024-09-13T11:55:00.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff339ffb640 10 monclient(hunting): _negotiate 2024-09-13T11:55:00.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff339ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:00.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff339ffb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:55:00.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff339ffb640 1 -- 172.21.15.100:0/2096276329 --> v1:172.21.15.122:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7ff32800a150 con 0x7ff3280034c0 2024-09-13T11:55:00.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff33a7fc640 10 monclient(hunting): handle_auth_done global_id 14124 payload 293 2024-09-13T11:55:00.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff33a7fc640 1 -- 172.21.15.100:0/2096276329 >> v1:172.21.15.122:6789/0 conn(0x7ff3280034c0 legacy=0x7ff328005940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:00.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff33a7fc640 10 monclient: _finish_hunting 0 2024-09-13T11:55:00.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff33a7fc640 1 monclient: found mon.noname-c 2024-09-13T11:55:00.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff33a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:00.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff33a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.122:3300/0 2024-09-13T11:55:00.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.059+0000 7ff33a7fc640 1 -- 172.21.15.100:0/2096276329 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff32800b3f0 con 0x7ff3280073c0 2024-09-13T11:55:00.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff33a7fc640 1 --2- 172.21.15.100:0/2096276329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3280073c0 0x7ff328009880 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff33c06a3c0 tx=0x7ff324009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=575abe80afb8747e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:00.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff339ffb640 1 -- 172.21.15.100:0/2096276329 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff324039070 con 0x7ff3280073c0 2024-09-13T11:55:00.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff339ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:00.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff339ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:55:00.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff339ffb640 10 monclient: dump: 2024-09-13T11:55:00.070 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:00.070 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:00.070 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:00.070 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:00.070 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:00.070 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:00.070 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:00.071 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:00.071 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:00.071 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:00.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff339ffb640 10 monclient: _finish_auth 0 2024-09-13T11:55:00.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff339ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:55:00.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff339ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:00.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff341975640 5 monclient: authenticate success, global_id 14124 2024-09-13T11:55:00.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff341975640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:55:00.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff339ffb640 1 -- 172.21.15.100:0/2096276329 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff324004230 con 0x7ff3280073c0 2024-09-13T11:55:00.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff339ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:55:00.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff341975640 10 monclient: get_monmap_and_config success 2024-09-13T11:55:00.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff341975640 4 set_mon_vals no callback set 2024-09-13T11:55:00.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff339ffb640 1 -- 172.21.15.100:0/2096276329 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff324032300 con 0x7ff3280073c0 2024-09-13T11:55:00.072 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff339ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:00.072 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff339ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:55:00.072 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff339ffb640 10 monclient: dump: 2024-09-13T11:55:00.072 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:00.072 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:00.072 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:00.072 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:00.072 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:00.072 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:00.072 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:00.072 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:00.072 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:00.072 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:00.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff341975640 10 monclient: shutdown 2024-09-13T11:55:00.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff341975640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:55:00.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff341975640 1 -- 172.21.15.100:0/2096276329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3280073c0 msgr2=0x7ff328009880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:00.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff341975640 1 --2- 172.21.15.100:0/2096276329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3280073c0 0x7ff328009880 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff33c06a3c0 tx=0x7ff324009190 comp rx=0 tx=0).stop 2024-09-13T11:55:00.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff33b7fe640 1 -- 172.21.15.100:0/2096276329 reap_dead start 2024-09-13T11:55:00.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff341975640 1 -- 172.21.15.100:0/2096276329 shutdown_connections 2024-09-13T11:55:00.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff341975640 1 --2- 172.21.15.100:0/2096276329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3280073c0 0x7ff328009880 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:00.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff341975640 1 -- 172.21.15.100:0/2096276329 >> 172.21.15.100:0/2096276329 conn(0x7ff33c0a68e0 msgr2=0x7ff33c0a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:55:00.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff341975640 1 -- 172.21.15.100:0/2096276329 shutdown_connections 2024-09-13T11:55:00.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.060+0000 7ff341975640 1 -- 172.21.15.100:0/2096276329 wait complete. 2024-09-13T11:55:00.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.061+0000 7ff341975640 10 monclient: build_initial_monmap 2024-09-13T11:55:00.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.061+0000 7ff341975640 10 monclient: monmap: 2024-09-13T11:55:00.074 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:55:00.074 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:55:00.074 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:55:00.074 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:55:00.074 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:55:00.074 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:00.074 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:55:00.074 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:55:00.074 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:55:00.074 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:00.074 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.061+0000 7ff341975640 1 Processor -- start 2024-09-13T11:55:00.074 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff341975640 1 -- start start 2024-09-13T11:55:00.074 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff341975640 10 monclient: init 2024-09-13T11:55:00.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff341975640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:00.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff341975640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:55:00.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff341975640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3280073c0 0x7ff33c09f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:00.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff341975640 10 monclient(hunting): picked mon.noname-c con 0x7ff3280073c0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:55:00.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff341975640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff33c120960 0x7ff33c1377d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:00.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff341975640 10 monclient(hunting): picked mon.noname-a con 0x7ff33c120960 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:55:00.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff341975640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff33c137d10 0x7ff33c1cc410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:00.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff341975640 10 monclient(hunting): picked mon.noname-b con 0x7ff33c137d10 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:55:00.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff341975640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:00.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff341975640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7ff33c0a5be0 con 0x7ff33c120960 2024-09-13T11:55:00.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff341975640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:00.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff341975640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff33c0a5a60 con 0x7ff33c137d10 2024-09-13T11:55:00.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff341975640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:00.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff341975640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7ff33c13f320 con 0x7ff3280073c0 2024-09-13T11:55:00.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff341975640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:00.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff341975640 10 monclient(hunting): authenticate will time out at 880.655533s 2024-09-13T11:55:00.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33affd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3280073c0 0x7ff33c09f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:00.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33affd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3280073c0 0x7ff33c09f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:36916/0 (socket says 172.21.15.100:36916) 2024-09-13T11:55:00.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33a7fc640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff33c120960 0x7ff33c1377d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:00.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33affd640 1 -- 172.21.15.100:0/1395406192 learned_addr learned my addr 172.21.15.100:0/1395406192 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:55:00.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33affd640 10 monclient(hunting): get_auth_request con 0x7ff3280073c0 auth_method 0 2024-09-13T11:55:00.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:00.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:00.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:00.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33a7fc640 10 monclient(hunting): get_auth_request con 0x7ff33c120960 auth_method 0 2024-09-13T11:55:00.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:00.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:00.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:00.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:00.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:00.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:00.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:00.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:00.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:00.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33b7fe640 1 --2- 172.21.15.100:0/1395406192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff33c137d10 0x7ff33c1cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:00.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33affd640 10 monclient(hunting): handle_auth_done global_id 4853 payload 824 2024-09-13T11:55:00.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33affd640 1 -- 172.21.15.100:0/1395406192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff33c137d10 msgr2=0x7ff33c1cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:00.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33affd640 1 --2- 172.21.15.100:0/1395406192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff33c137d10 0x7ff33c1cc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:00.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33affd640 1 -- 172.21.15.100:0/1395406192 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff33c120960 msgr2=0x7ff33c1377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:00.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33affd640 1 --2- 172.21.15.100:0/1395406192 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff33c120960 0x7ff33c1377d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:00.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33affd640 10 monclient: _finish_hunting 0 2024-09-13T11:55:00.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33affd640 1 monclient: found mon.noname-c 2024-09-13T11:55:00.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:00.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:55:00.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33affd640 1 -- 172.21.15.100:0/1395406192 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff33c138470 con 0x7ff3280073c0 2024-09-13T11:55:00.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33b7fe640 10 monclient: get_auth_request con 0x7ff33c137d10 auth_method 0 2024-09-13T11:55:00.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33a7fc640 1 --2- 172.21.15.100:0/1395406192 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff33c120960 0x7ff33c1377d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:55:00.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.062+0000 7ff33b7fe640 1 --2- 172.21.15.100:0/1395406192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff33c137d10 0x7ff33c1cc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:55:00.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.063+0000 7ff33affd640 1 --2- 172.21.15.100:0/1395406192 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3280073c0 0x7ff33c09f080 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff33400c890 tx=0x7ff33400cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:00.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.063+0000 7ff333fff640 1 -- 172.21.15.100:0/1395406192 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33400a640 con 0x7ff3280073c0 2024-09-13T11:55:00.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.063+0000 7ff333fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:00.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.063+0000 7ff333fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:55:00.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.063+0000 7ff333fff640 10 monclient: dump: 2024-09-13T11:55:00.079 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:00.079 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:00.079 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:00.079 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:00.079 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:00.079 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:00.079 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:00.079 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:00.079 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:00.079 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:00.079 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.063+0000 7ff333fff640 10 monclient: _finish_auth 0 2024-09-13T11:55:00.079 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.063+0000 7ff333fff640 10 monclient: _check_auth_tickets 2024-09-13T11:55:00.079 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.063+0000 7ff333fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:00.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.063+0000 7ff341975640 5 monclient: authenticate success, global_id 4853 2024-09-13T11:55:00.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.063+0000 7ff333fff640 1 -- 172.21.15.100:0/1395406192 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff33400a7e0 con 0x7ff3280073c0 2024-09-13T11:55:00.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.063+0000 7ff341975640 10 monclient: _renew_subs 2024-09-13T11:55:00.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.063+0000 7ff341975640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:55:00.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.063+0000 7ff341975640 1 -- 172.21.15.100:0/1395406192 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff33c0a10d0 con 0x7ff3280073c0 2024-09-13T11:55:00.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.063+0000 7ff333fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:55:00.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.063+0000 7ff333fff640 1 -- 172.21.15.100:0/1395406192 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33400abd0 con 0x7ff3280073c0 2024-09-13T11:55:00.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.063+0000 7ff333fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:00.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.063+0000 7ff333fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:55:00.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.063+0000 7ff333fff640 10 monclient: dump: 2024-09-13T11:55:00.080 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:00.080 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:00.080 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:00.081 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:00.081 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:00.081 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:00.081 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:00.081 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:00.081 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:00.081 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:00.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.063+0000 7ff341975640 10 monclient: _renew_subs 2024-09-13T11:55:00.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.063+0000 7ff3397fa640 4 set_mon_vals no callback set 2024-09-13T11:55:00.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.063+0000 7ff341975640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:55:00.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.063+0000 7ff341975640 1 -- 172.21.15.100:0/1395406192 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff33c121e80 con 0x7ff3280073c0 2024-09-13T11:55:00.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.064+0000 7ff333fff640 1 -- 172.21.15.100:0/1395406192 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7ff334011070 con 0x7ff3280073c0 2024-09-13T11:55:00.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.064+0000 7ff333fff640 1 --2- 172.21.15.100:0/1395406192 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff31c03dfd0 0x7ff31c040490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:00.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.064+0000 7ff341975640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:00.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.064+0000 7ff341975640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:00.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.064+0000 7ff341975640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:55:00.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.064+0000 7ff333fff640 1 -- 172.21.15.100:0/1395406192 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(113..113 src has 1..113) ==== 4738+0+0 (secure 0 0 0) 0x7ff3340520a0 con 0x7ff3280073c0 2024-09-13T11:55:00.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.064+0000 7ff341975640 1 -- 172.21.15.100:0/1395406192 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff33c06ee30 con 0x7ff3280073c0 2024-09-13T11:55:00.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.064+0000 7ff33a7fc640 1 --2- 172.21.15.100:0/1395406192 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff31c03dfd0 0x7ff31c040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:00.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.064+0000 7ff33a7fc640 10 monclient: get_auth_request con 0x7ff31c03dfd0 auth_method 0 2024-09-13T11:55:00.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.065+0000 7ff33a7fc640 1 --2- 172.21.15.100:0/1395406192 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff31c03dfd0 0x7ff31c040490 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff328003470 tx=0x7ff324041040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:00.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.069+0000 7ff333fff640 1 -- 172.21.15.100:0/1395406192 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7ff33c06fa80 con 0x7ff3280073c0 2024-09-13T11:55:00.083 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.069+0000 7ff333fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:00.083 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.069+0000 7ff333fff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:55:00.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.308+0000 7ff341975640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-13T11:55:00.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.308+0000 7ff341975640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-13T11:55:00.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.308+0000 7ff341975640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:55:00.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.308+0000 7ff341975640 1 -- 172.21.15.100:0/1395406192 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7ff33c0a0050 con 0x7ff3280073c0 2024-09-13T11:55:00.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.309+0000 7ff333fff640 1 -- 172.21.15.100:0/1395406192 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7ff3340078c0 con 0x7ff3280073c0 2024-09-13T11:55:00.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.309+0000 7ff333fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-13T11:55:00.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.309+0000 7ff333fff640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:55:00.311 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.309+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:55:00.311 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.309+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=7,config=2,health=11,kv=11,logm=69} crc {auth=229840821,config=1097430493,health=3487487429,kv=2029344140,logm=715124297}) 2024-09-13T11:55:00.311 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.309+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=8,config=2,health=12,kv=12,logm=66} crc {auth=505881978,config=1097430493,health=306129551,kv=4190740400,logm=1780488051}) 2024-09-13T11:55:00.311 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.309+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:55:00.311 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.309+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=7,config=2,health=11,kv=11,logm=69} crc {auth=229840821,config=1097430493,health=3487487429,kv=2029344140,logm=715124297}) 2024-09-13T11:55:00.311 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.309+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=7,config=2,health=11,kv=12,logm=68} crc {auth=2722142207,config=1097430493,health=1717611308,kv=3786135054,logm=1395135766}) 2024-09-13T11:55:00.312 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.310+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:55:00.312 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.310+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2088759840}) 2024-09-13T11:55:00.312 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.310+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2963427312}) 2024-09-13T11:55:00.312 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.310+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:55:00.312 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.310+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2088759840}) 2024-09-13T11:55:00.312 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.310+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3119761318}) 2024-09-13T11:55:00.313 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.311+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:55:00.313 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.311+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3634558932}) 2024-09-13T11:55:00.313 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.311+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=107597118}) 2024-09-13T11:55:00.313 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.311+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:55:00.313 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.311+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3634558932}) 2024-09-13T11:55:00.313 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.311+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322806764}) 2024-09-13T11:55:00.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.312+0000 7ff341975640 1 -- 172.21.15.100:0/1395406192 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff31c03dfd0 msgr2=0x7ff31c040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:00.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.312+0000 7ff341975640 1 --2- 172.21.15.100:0/1395406192 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff31c03dfd0 0x7ff31c040490 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff328003470 tx=0x7ff324041040 comp rx=0 tx=0).stop 2024-09-13T11:55:00.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.312+0000 7ff341975640 10 monclient: shutdown 2024-09-13T11:55:00.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.312+0000 7ff341975640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:55:00.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.312+0000 7ff341975640 1 -- 172.21.15.100:0/1395406192 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3280073c0 msgr2=0x7ff33c09f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:00.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.312+0000 7ff341975640 1 --2- 172.21.15.100:0/1395406192 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3280073c0 0x7ff33c09f080 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff33400c890 tx=0x7ff33400cd50 comp rx=0 tx=0).stop 2024-09-13T11:55:00.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.312+0000 7ff341975640 1 -- 172.21.15.100:0/1395406192 shutdown_connections 2024-09-13T11:55:00.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.312+0000 7ff341975640 1 --2- 172.21.15.100:0/1395406192 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff31c03dfd0 0x7ff31c040490 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:00.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.312+0000 7ff341975640 1 --2- 172.21.15.100:0/1395406192 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff33c120960 0x7ff33c1377d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:00.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.312+0000 7ff341975640 1 --2- 172.21.15.100:0/1395406192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff33c137d10 0x7ff33c1cc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:00.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.312+0000 7ff341975640 1 --2- 172.21.15.100:0/1395406192 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3280073c0 0x7ff33c09f080 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:00.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.312+0000 7ff341975640 1 -- 172.21.15.100:0/1395406192 >> 172.21.15.100:0/1395406192 conn(0x7ff33c0a68e0 msgr2=0x7ff33c13dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:55:00.315 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.312+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:55:00.316 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.312+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4050056371}) 2024-09-13T11:55:00.316 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.312+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3930865086}) 2024-09-13T11:55:00.316 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.312+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:55:00.316 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.312+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4050056371}) 2024-09-13T11:55:00.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.312+0000 7ff341975640 1 -- 172.21.15.100:0/1395406192 shutdown_connections 2024-09-13T11:55:00.316 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.312+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1090185362}) 2024-09-13T11:55:00.317 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:00.312+0000 7ff341975640 1 -- 172.21.15.100:0/1395406192 wait complete. 2024-09-13T11:55:00.318 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.313+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:55:00.318 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.313+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2606258541}) 2024-09-13T11:55:00.318 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.313+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1743740176}) 2024-09-13T11:55:00.318 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.313+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:55:00.319 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.313+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2606258541}) 2024-09-13T11:55:00.319 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.313+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4083685781}) 2024-09-13T11:55:00.319 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.314+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:55:00.319 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.314+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2694886385}) 2024-09-13T11:55:00.319 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.314+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2441878472}) 2024-09-13T11:55:00.319 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.314+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:55:00.319 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.314+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2694886385}) 2024-09-13T11:55:00.319 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.314+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=98,mdsmap=2} crc {logm=2195214741,mdsmap=1089634524}) 2024-09-13T11:55:00.319 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.315+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:55:00.319 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.315+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=3,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=19,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=41} crc {logm=4023742920,mdsmap=3458900325,mgr=1699291536,mgr_command_descs=3192186153,mgr_metadata=4154780615,mgrstat=2938752899,mon_config_key=3555263807,monmap=157995306,nvmeofgw=1596038039,osd_metadata=789425444,osd_pg_creating=1402940953,osd_snap=841190635,osdmap=3604150088}) 2024-09-13T11:55:00.319 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.315+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=3,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=18,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=41} crc {logm=4023742920,mdsmap=3140514010,mgr=4016471625,mgr_command_descs=3192186153,mgr_metadata=4154780615,mgrstat=1124626721,mon_config_key=3392347370,monmap=157995306,nvmeofgw=598337686,osd_metadata=3051479690,osd_pg_creating=1402940953,osd_snap=841190635,osdmap=2271523937}) 2024-09-13T11:55:00.319 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.315+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:55:00.319 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.315+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=3,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=19,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=41} crc {logm=4023742920,mdsmap=3458900325,mgr=1699291536,mgr_command_descs=3192186153,mgr_metadata=4154780615,mgrstat=2938752899,mon_config_key=3555263807,monmap=157995306,nvmeofgw=1596038039,osd_metadata=789425444,osd_pg_creating=1402940953,osd_snap=841190635,osdmap=3604150088}) 2024-09-13T11:55:00.320 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.315+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=12,mon_config_key=18,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=41} crc {logm=4023742920,mdsmap=3458900325,mgr=2459773887,mgr_metadata=4154780615,mgrstat=1124626721,mon_config_key=4104984287,monmap=157995306,nvmeofgw=598337686,osd_metadata=1585186621,osd_pg_creating=1402940953,osd_snap=841190635,osdmap=3477132792}) 2024-09-13T11:55:00.321 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.319+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:55:00.321 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.319+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2937070047}) 2024-09-13T11:55:00.321 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.319+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1189450072}) 2024-09-13T11:55:00.321 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.319+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:55:00.321 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.319+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2937070047}) 2024-09-13T11:55:00.321 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.319+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2067212687}) 2024-09-13T11:55:00.330 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-13T11:55:00.331 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.329+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:55:00.332 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.329+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=41} crc {osdmap=3848057241}) 2024-09-13T11:55:00.332 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.329+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=36} crc {osdmap=1061365495}) 2024-09-13T11:55:00.332 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.329+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:55:00.332 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.329+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=41} crc {osdmap=3848057241}) 2024-09-13T11:55:00.332 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:55:00.329+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=39} crc {osdmap=2609919641}) 2024-09-13T11:55:00.400 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:00.400 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36193]: starting. 2024-09-13T11:55:00.400 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36193]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-36156 2024-09-13T11:55:00.401 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36193]: created object 0... 2024-09-13T11:55:00.401 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36193]: created object 25... 2024-09-13T11:55:00.401 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36193]: created object 49... 2024-09-13T11:55:00.401 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36193]: finishing. 2024-09-13T11:55:00.401 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36193]: shutting down. 2024-09-13T11:55:00.614 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:00.615 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36195]: starting. 2024-09-13T11:55:00.615 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36195]: listing objects. 2024-09-13T11:55:00.615 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36195]: listed object 0... 2024-09-13T11:55:00.615 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36195]: listed object 25... 2024-09-13T11:55:00.615 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36195]: saw 50 objects 2024-09-13T11:55:00.615 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36195]: shutting down. 2024-09-13T11:55:01.330 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-13T11:55:01.331 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:55:01.340 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:01.340 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36194]: starting. 2024-09-13T11:55:01.341 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36194]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-36156 2024-09-13T11:55:01.341 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36194]: shutting down. 2024-09-13T11:55:01.342 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:01.342 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:01.342 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:55:01.344 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:55:01.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.437+0000 7fb57eb4b640 10 monclient: get_monmap_and_config 2024-09-13T11:55:01.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.437+0000 7fb57eb4b640 10 monclient: build_initial_monmap 2024-09-13T11:55:01.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.437+0000 7fb57eb4b640 10 monclient: monmap: 2024-09-13T11:55:01.439 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:55:01.439 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:01.439 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:55:01.438396+0000 2024-09-13T11:55:01.439 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:55:01.438396+0000 2024-09-13T11:55:01.439 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:55:01.439 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:01.439 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:55:01.439 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:55:01.439 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:55:01.439 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:55:01.439 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:55:01.440 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:01.448 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.438+0000 7fb57eb4b640 1 Processor -- start 2024-09-13T11:55:01.448 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.438+0000 7fb57eb4b640 1 -- start start 2024-09-13T11:55:01.448 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.438+0000 7fb57eb4b640 10 monclient: init 2024-09-13T11:55:01.448 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.438+0000 7fb57eb4b640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:01.448 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.438+0000 7fb57eb4b640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-13T11:55:01.448 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.438+0000 7fb57eb4b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5780a4870 addr v1:172.21.15.100:6789/0 2024-09-13T11:55:01.448 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.438+0000 7fb57eb4b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5780a5360 0x7fb57813ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:01.448 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.438+0000 7fb57eb4b640 10 monclient(hunting): picked mon.noname-b con 0x7fb5780a5360 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:55:01.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.438+0000 7fb57eb4b640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb57813cfc0 0x7fb57813f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:01.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.438+0000 7fb57eb4b640 10 monclient(hunting): picked mon.noname-a con 0x7fb57813cfc0 addr v2:172.21.15.100:3300/0 2024-09-13T11:55:01.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.438+0000 7fb57eb4b640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5780701d0 con 0x7fb5780a4870 2024-09-13T11:55:01.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.438+0000 7fb57eb4b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:01.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.438+0000 7fb57eb4b640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7fb5780a1df0 con 0x7fb57813cfc0 2024-09-13T11:55:01.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.438+0000 7fb57eb4b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:01.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.438+0000 7fb57eb4b640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fb5780a5960 con 0x7fb5780a5360 2024-09-13T11:55:01.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.438+0000 7fb57eb4b640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:01.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.438+0000 7fb57eb4b640 10 monclient(hunting): authenticate will time out at 882.031939s 2024-09-13T11:55:01.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb577fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5780a5360 0x7fb57813ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:01.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb577fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5780a5360 0x7fb57813ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:36930/0 (socket says 172.21.15.100:36930) 2024-09-13T11:55:01.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb57c8e7640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fb5780a4870 0x7fb5780a4c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:48258/0 (socket says 172.21.15.100:48258) 2024-09-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb577fff640 1 -- 172.21.15.100:0/2105981154 learned_addr learned my addr 172.21.15.100:0/2105981154 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb577fff640 10 monclient(hunting): get_auth_request con 0x7fb5780a5360 auth_method 0 2024-09-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb577fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb577fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb577fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb57d0e8640 1 --2- 172.21.15.100:0/2105981154 >> v2:172.21.15.100:3300/0 conn(0x7fb57813cfc0 0x7fb57813f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb577fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb577fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb577fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb57d0e8640 10 monclient(hunting): get_auth_request con 0x7fb57813cfc0 auth_method 0 2024-09-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb57d0e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb57d0e8640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb57d0e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb57d0e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:01.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb57d0e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:01.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb57d0e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:01.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb577fff640 10 monclient(hunting): handle_auth_done global_id 4865 payload 274 2024-09-13T11:55:01.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb577fff640 1 -- 172.21.15.100:0/2105981154 >> v2:172.21.15.100:3300/0 conn(0x7fb57813cfc0 msgr2=0x7fb57813f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:55:01.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb577fff640 1 --2- 172.21.15.100:0/2105981154 >> v2:172.21.15.100:3300/0 conn(0x7fb57813cfc0 0x7fb57813f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:01.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb577fff640 1 -- 172.21.15.100:0/2105981154 >> v1:172.21.15.100:6789/0 conn(0x7fb5780a4870 legacy=0x7fb5780a4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:01.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb577fff640 10 monclient: _finish_hunting 0 2024-09-13T11:55:01.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb577fff640 1 monclient: found mon.noname-b 2024-09-13T11:55:01.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb577fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:01.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb577fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:55:01.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb577fff640 1 -- 172.21.15.100:0/2105981154 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5780a0ed0 con 0x7fb5780a5360 2024-09-13T11:55:01.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb5777fe640 1 -- 172.21.15.100:0/2105981154 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 640703450 0 0) 0x7fb5780701d0 con 0x7fb5780a4870 2024-09-13T11:55:01.451 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.439+0000 7fb5777fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:55:01.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb577fff640 1 --2- 172.21.15.100:0/2105981154 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5780a5360 0x7fb57813ca80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb564009870 tx=0x7fb56402a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=973cc4e15052b45c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:01.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb5777fe640 1 -- 172.21.15.100:0/2105981154 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb564037070 con 0x7fb5780a5360 2024-09-13T11:55:01.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb5777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:01.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb5777fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:55:01.452 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb5777fe640 10 monclient: dump: 2024-09-13T11:55:01.452 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:01.452 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:01.452 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.452 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.452 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:01.452 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:01.452 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:01.452 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:01.453 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:01.453 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:01.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb5777fe640 10 monclient: _finish_auth 0 2024-09-13T11:55:01.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb5777fe640 10 monclient: _check_auth_tickets 2024-09-13T11:55:01.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb5777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:01.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb5777fe640 1 -- 172.21.15.100:0/2105981154 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb564032440 con 0x7fb5780a5360 2024-09-13T11:55:01.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb5777fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:55:01.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb5777fe640 1 -- 172.21.15.100:0/2105981154 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb564032810 con 0x7fb5780a5360 2024-09-13T11:55:01.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb5777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:01.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb5777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:55:01.453 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb5777fe640 10 monclient: dump: 2024-09-13T11:55:01.453 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:01.453 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:01.454 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.454 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.454 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:01.454 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:01.454 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:01.454 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:01.454 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:01.454 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:01.454 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb57eb4b640 5 monclient: authenticate success, global_id 4865 2024-09-13T11:55:01.454 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb57eb4b640 10 monclient: get_monmap_and_config success 2024-09-13T11:55:01.454 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb57eb4b640 4 set_mon_vals no callback set 2024-09-13T11:55:01.454 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb57eb4b640 10 monclient: shutdown 2024-09-13T11:55:01.454 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb57eb4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:55:01.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb57eb4b640 1 -- 172.21.15.100:0/2105981154 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5780a5360 msgr2=0x7fb57813ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:01.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb57eb4b640 1 --2- 172.21.15.100:0/2105981154 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5780a5360 0x7fb57813ca80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb564009870 tx=0x7fb56402a130 comp rx=0 tx=0).stop 2024-09-13T11:55:01.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb57eb4b640 1 -- 172.21.15.100:0/2105981154 shutdown_connections 2024-09-13T11:55:01.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb57eb4b640 1 --2- 172.21.15.100:0/2105981154 >> v2:172.21.15.100:3300/0 conn(0x7fb57813cfc0 0x7fb57813f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:01.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb57eb4b640 1 --2- 172.21.15.100:0/2105981154 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5780a5360 0x7fb57813ca80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:01.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb57eb4b640 1 -- 172.21.15.100:0/2105981154 >> 172.21.15.100:0/2105981154 conn(0x7fb5780a6970 msgr2=0x7fb5780a6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:55:01.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb57eb4b640 1 -- 172.21.15.100:0/2105981154 shutdown_connections 2024-09-13T11:55:01.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.440+0000 7fb57eb4b640 1 -- 172.21.15.100:0/2105981154 wait complete. 2024-09-13T11:55:01.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.441+0000 7fb57eb4b640 10 monclient: build_initial_monmap 2024-09-13T11:55:01.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.441+0000 7fb57eb4b640 10 monclient: monmap: 2024-09-13T11:55:01.455 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:55:01.455 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:55:01.455 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:55:01.455 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:55:01.456 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:55:01.456 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:01.456 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:55:01.456 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:55:01.456 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:55:01.456 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:01.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.441+0000 7fb57eb4b640 1 Processor -- start 2024-09-13T11:55:01.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.441+0000 7fb57eb4b640 1 -- start start 2024-09-13T11:55:01.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.441+0000 7fb57eb4b640 10 monclient: init 2024-09-13T11:55:01.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57eb4b640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:01.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57eb4b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:55:01.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57eb4b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5780a4870 0x7fb578121bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:01.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57eb4b640 10 monclient(hunting): picked mon.noname-b con 0x7fb5780a4870 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:55:01.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57eb4b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5780a5360 0x7fb5781cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:01.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57eb4b640 10 monclient(hunting): picked mon.noname-c con 0x7fb5780a5360 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:55:01.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57eb4b640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb57813cfc0 0x7fb578136a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:01.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57eb4b640 10 monclient(hunting): picked mon.noname-a con 0x7fb57813cfc0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:55:01.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57eb4b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:01.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57eb4b640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fb57806fde0 con 0x7fb57813cfc0 2024-09-13T11:55:01.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57eb4b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:01.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57eb4b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb57806fc60 con 0x7fb5780a4870 2024-09-13T11:55:01.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57eb4b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:01.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57eb4b640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fb5780a27d0 con 0x7fb5780a5360 2024-09-13T11:55:01.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57eb4b640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:01.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57eb4b640 10 monclient(hunting): authenticate will time out at 882.035462s 2024-09-13T11:55:01.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb577fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5780a5360 0x7fb5781cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:01.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57d0e8640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb57813cfc0 0x7fb578136a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:01.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb577fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5780a5360 0x7fb5781cc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:36934/0 (socket says 172.21.15.100:36934) 2024-09-13T11:55:01.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb577fff640 1 -- 172.21.15.100:0/3999909288 learned_addr learned my addr 172.21.15.100:0/3999909288 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:55:01.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb577fff640 10 monclient(hunting): get_auth_request con 0x7fb5780a5360 auth_method 0 2024-09-13T11:55:01.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57d0e8640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb57813cfc0 0x7fb578136a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53462/0 (socket says 172.21.15.100:53462) 2024-09-13T11:55:01.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb577fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:01.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb577fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:01.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb577fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:01.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57d0e8640 10 monclient(hunting): get_auth_request con 0x7fb57813cfc0 auth_method 0 2024-09-13T11:55:01.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57d0e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:01.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57d0e8640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:01.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57d0e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:01.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb577fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:01.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb577fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:01.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb577fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:01.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57d0e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:01.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57d0e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:01.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57d0e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:01.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57c8e7640 1 --2- 172.21.15.100:0/3999909288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5780a4870 0x7fb578121bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:01.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57c8e7640 10 monclient(hunting): get_auth_request con 0x7fb5780a4870 auth_method 0 2024-09-13T11:55:01.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57c8e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:01.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57c8e7640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:01.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57c8e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:01.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57c8e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:01.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57c8e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:01.459 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57c8e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:01.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb577fff640 10 monclient(hunting): handle_auth_done global_id 4868 payload 824 2024-09-13T11:55:01.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb577fff640 1 -- 172.21.15.100:0/3999909288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5780a4870 msgr2=0x7fb578121bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:01.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb577fff640 1 --2- 172.21.15.100:0/3999909288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5780a4870 0x7fb578121bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:01.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb577fff640 1 -- 172.21.15.100:0/3999909288 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb57813cfc0 msgr2=0x7fb578136a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:01.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb577fff640 1 --2- 172.21.15.100:0/3999909288 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb57813cfc0 0x7fb578136a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:01.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb577fff640 10 monclient: _finish_hunting 0 2024-09-13T11:55:01.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb577fff640 1 monclient: found mon.noname-c 2024-09-13T11:55:01.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb577fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:01.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb577fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:55:01.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb577fff640 1 -- 172.21.15.100:0/3999909288 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5780a10d0 con 0x7fb5780a5360 2024-09-13T11:55:01.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.442+0000 7fb57d0e8640 1 --2- 172.21.15.100:0/3999909288 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb57813cfc0 0x7fb578136a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:55:01.460 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.443+0000 7fb577fff640 1 --2- 172.21.15.100:0/3999909288 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5780a5360 0x7fb5781cc2c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb564000c00 tx=0x7fb5640046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:01.461 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.443+0000 7fb5757fa640 1 -- 172.21.15.100:0/3999909288 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb564041020 con 0x7fb5780a5360 2024-09-13T11:55:01.461 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.443+0000 7fb5757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:01.461 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.443+0000 7fb5757fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:55:01.461 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.443+0000 7fb5757fa640 10 monclient: dump: 2024-09-13T11:55:01.461 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:01.461 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:01.461 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.461 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.461 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:01.461 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:01.461 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:01.461 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:01.461 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:01.462 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:01.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.443+0000 7fb5757fa640 10 monclient: _finish_auth 0 2024-09-13T11:55:01.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.443+0000 7fb5757fa640 10 monclient: _check_auth_tickets 2024-09-13T11:55:01.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.443+0000 7fb5757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:01.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.443+0000 7fb57eb4b640 5 monclient: authenticate success, global_id 4868 2024-09-13T11:55:01.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.443+0000 7fb5757fa640 1 -- 172.21.15.100:0/3999909288 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb564030640 con 0x7fb5780a5360 2024-09-13T11:55:01.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.443+0000 7fb57eb4b640 10 monclient: _renew_subs 2024-09-13T11:55:01.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.443+0000 7fb57eb4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:55:01.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.443+0000 7fb57eb4b640 1 -- 172.21.15.100:0/3999909288 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5780a0a50 con 0x7fb5780a5360 2024-09-13T11:55:01.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.443+0000 7fb5757fa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:55:01.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.443+0000 7fb5757fa640 1 -- 172.21.15.100:0/3999909288 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb564037040 con 0x7fb5780a5360 2024-09-13T11:55:01.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.443+0000 7fb5757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:01.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.443+0000 7fb5757fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:55:01.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.443+0000 7fb5757fa640 10 monclient: dump: 2024-09-13T11:55:01.463 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:01.463 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:01.463 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.463 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.463 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:01.463 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:01.463 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:01.463 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:01.463 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:01.463 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:01.463 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.443+0000 7fb57eb4b640 10 monclient: _renew_subs 2024-09-13T11:55:01.463 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.443+0000 7fb57eb4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:55:01.463 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.443+0000 7fb57eb4b640 1 -- 172.21.15.100:0/3999909288 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb578136f60 con 0x7fb5780a5360 2024-09-13T11:55:01.464 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.443+0000 7fb576ffd640 4 set_mon_vals no callback set 2024-09-13T11:55:01.464 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.444+0000 7fb5757fa640 1 -- 172.21.15.100:0/3999909288 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fb564044020 con 0x7fb5780a5360 2024-09-13T11:55:01.464 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.444+0000 7fb57eb4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:01.464 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.444+0000 7fb57eb4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:01.464 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.444+0000 7fb57eb4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:55:01.464 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.444+0000 7fb57eb4b640 1 -- 172.21.15.100:0/3999909288 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5780a4c50 con 0x7fb5780a5360 2024-09-13T11:55:01.464 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.444+0000 7fb5757fa640 1 --2- 172.21.15.100:0/3999909288 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fb55c03dfa0 0x7fb55c040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:01.464 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.444+0000 7fb5757fa640 1 -- 172.21.15.100:0/3999909288 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(115..115 src has 1..115) ==== 4346+0+0 (secure 0 0 0) 0x7fb564072ce0 con 0x7fb5780a5360 2024-09-13T11:55:01.464 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.444+0000 7fb57c8e7640 1 --2- 172.21.15.100:0/3999909288 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fb55c03dfa0 0x7fb55c040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:01.464 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.444+0000 7fb57c8e7640 10 monclient: get_auth_request con 0x7fb55c03dfa0 auth_method 0 2024-09-13T11:55:01.464 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.445+0000 7fb57c8e7640 1 --2- 172.21.15.100:0/3999909288 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fb55c03dfa0 0x7fb55c040460 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb56c000f60 tx=0x7fb56c007cf0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:01.465 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.449+0000 7fb5757fa640 1 -- 172.21.15.100:0/3999909288 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fb56404b030 con 0x7fb5780a5360 2024-09-13T11:55:01.465 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.449+0000 7fb5757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:01.465 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.449+0000 7fb5757fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:55:01.694 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.692+0000 7fb57eb4b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:55:01.694 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.692+0000 7fb57eb4b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:55:01.694 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.692+0000 7fb57eb4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:55:01.695 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.692+0000 7fb57eb4b640 1 -- 172.21.15.100:0/3999909288 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb578123330 con 0x7fb5780a5360 2024-09-13T11:55:01.695 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.693+0000 7fb5757fa640 1 -- 172.21.15.100:0/3999909288 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fb56403c380 con 0x7fb5780a5360 2024-09-13T11:55:01.695 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.693+0000 7fb5757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:55:01.695 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.693+0000 7fb5757fa640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:55:01.695 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:55:01.697 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.695+0000 7fb57eb4b640 1 -- 172.21.15.100:0/3999909288 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fb55c03dfa0 msgr2=0x7fb55c040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:01.697 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.695+0000 7fb57eb4b640 1 --2- 172.21.15.100:0/3999909288 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fb55c03dfa0 0x7fb55c040460 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb56c000f60 tx=0x7fb56c007cf0 comp rx=0 tx=0).stop 2024-09-13T11:55:01.697 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.695+0000 7fb57eb4b640 10 monclient: shutdown 2024-09-13T11:55:01.697 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.695+0000 7fb57eb4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:55:01.697 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.695+0000 7fb57eb4b640 1 -- 172.21.15.100:0/3999909288 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5780a5360 msgr2=0x7fb5781cc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:01.697 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.695+0000 7fb57eb4b640 1 --2- 172.21.15.100:0/3999909288 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5780a5360 0x7fb5781cc2c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb564000c00 tx=0x7fb5640046c0 comp rx=0 tx=0).stop 2024-09-13T11:55:01.697 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.695+0000 7fb57eb4b640 1 -- 172.21.15.100:0/3999909288 shutdown_connections 2024-09-13T11:55:01.697 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.695+0000 7fb57eb4b640 1 --2- 172.21.15.100:0/3999909288 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fb55c03dfa0 0x7fb55c040460 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:01.698 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.695+0000 7fb57eb4b640 1 --2- 172.21.15.100:0/3999909288 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb57813cfc0 0x7fb578136a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:01.698 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.695+0000 7fb57eb4b640 1 --2- 172.21.15.100:0/3999909288 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5780a5360 0x7fb5781cc2c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:01.698 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.695+0000 7fb57eb4b640 1 --2- 172.21.15.100:0/3999909288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5780a4870 0x7fb578121bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:01.698 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.695+0000 7fb57eb4b640 1 -- 172.21.15.100:0/3999909288 >> 172.21.15.100:0/3999909288 conn(0x7fb5780a6970 msgr2=0x7fb5780a8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:55:01.698 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.695+0000 7fb57eb4b640 1 -- 172.21.15.100:0/3999909288 shutdown_connections 2024-09-13T11:55:01.698 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.695+0000 7fb57eb4b640 1 -- 172.21.15.100:0/3999909288 wait complete. 2024-09-13T11:55:01.707 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-13T11:55:01.707 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-13T11:55:01.707 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-13T11:55:01.707 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-13T11:55:01.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.804+0000 7f2fb13f8640 10 monclient: get_monmap_and_config 2024-09-13T11:55:01.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.804+0000 7f2fb13f8640 10 monclient: build_initial_monmap 2024-09-13T11:55:01.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.804+0000 7f2fb13f8640 10 monclient: monmap: 2024-09-13T11:55:01.806 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:55:01.806 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:01.806 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:55:01.804986+0000 2024-09-13T11:55:01.806 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:55:01.804986+0000 2024-09-13T11:55:01.806 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:55:01.806 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:01.806 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:55:01.806 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:55:01.806 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:55:01.806 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:55:01.807 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:55:01.807 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:01.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.804+0000 7f2fb13f8640 1 Processor -- start 2024-09-13T11:55:01.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.804+0000 7f2fb13f8640 1 -- start start 2024-09-13T11:55:01.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.804+0000 7f2fb13f8640 10 monclient: init 2024-09-13T11:55:01.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2fb13f8640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:01.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2fb13f8640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-13T11:55:01.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2fb13f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2fac06ea50 addr v1:172.21.15.122:6789/0 2024-09-13T11:55:01.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2fb13f8640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2fac0a2860 0x7f2fac06f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:01.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2fb13f8640 10 monclient(hunting): picked mon.noname-c con 0x7f2fac0a2860 addr v2:172.21.15.122:3300/0 2024-09-13T11:55:01.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2fb13f8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2fac13c9f0 addr v1:172.21.15.100:6789/0 2024-09-13T11:55:01.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2fb13f8640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fac06fa80 con 0x7f2fac13c9f0 2024-09-13T11:55:01.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2fb13f8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fac0a1590 con 0x7f2fac06ea50 2024-09-13T11:55:01.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2fb13f8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:01.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2fb13f8640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f2fac0a2cb0 con 0x7f2fac0a2860 2024-09-13T11:55:01.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2fb13f8640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:01.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2fb13f8640 10 monclient(hunting): authenticate will time out at 882.398418s 2024-09-13T11:55:01.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2fab7fe640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f2fac13c9f0 0x7f2fac13ede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:48262/0 (socket says 172.21.15.100:48262) 2024-09-13T11:55:01.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2fab7fe640 1 -- 172.21.15.100:0/1900901656 learned_addr learned my addr 172.21.15.100:0/1900901656 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:55:01.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2fa3fff640 1 --2- 172.21.15.100:0/1900901656 >> v2:172.21.15.122:3300/0 conn(0x7f2fac0a2860 0x7f2fac06f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:01.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2faa7fc640 1 -- 172.21.15.100:0/1900901656 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1542690126 0 0) 0x7f2fa4001490 con 0x7f2fac13c9f0 2024-09-13T11:55:01.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2faa7fc640 10 monclient(hunting): _negotiate 2024-09-13T11:55:01.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2faa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:01.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2faa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:01.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2faa7fc640 10 monclient(hunting): my global_id is 14160 2024-09-13T11:55:01.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2faa7fc640 1 -- 172.21.15.100:0/1900901656 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f98003520 con 0x7f2fac13c9f0 2024-09-13T11:55:01.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2fa3fff640 10 monclient(hunting): get_auth_request con 0x7f2fac0a2860 auth_method 0 2024-09-13T11:55:01.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2fa3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:01.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2fa3fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:01.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2fa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:01.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2fa3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:01.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2fa3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:01.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2fa3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:01.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2faa7fc640 1 -- 172.21.15.100:0/1900901656 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 857033157 0 0) 0x7f2f98003520 con 0x7f2fac13c9f0 2024-09-13T11:55:01.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2faa7fc640 1 -- 172.21.15.100:0/1900901656 >> v2:172.21.15.122:3300/0 conn(0x7f2fac0a2860 msgr2=0x7f2fac06f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:55:01.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2faa7fc640 1 --2- 172.21.15.100:0/1900901656 >> v2:172.21.15.122:3300/0 conn(0x7f2fac0a2860 0x7f2fac06f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:01.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2faa7fc640 1 -- 172.21.15.100:0/1900901656 >> v1:172.21.15.122:6789/0 conn(0x7f2fac06ea50 legacy=0x7f2fac06ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:01.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2faa7fc640 10 monclient: _finish_hunting 0 2024-09-13T11:55:01.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2faa7fc640 1 monclient: found mon.noname-a-legacy 2024-09-13T11:55:01.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2faa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:01.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2faa7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.100:6789/0 2024-09-13T11:55:01.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2faa7fc640 1 -- 172.21.15.100:0/1900901656 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fac0a1870 con 0x7f2fac13c9f0 2024-09-13T11:55:01.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2faa7fc640 10 monclient: _finish_auth 0 2024-09-13T11:55:01.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2faa7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:55:01.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2faa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:01.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2fb13f8640 5 monclient: authenticate success, global_id 14160 2024-09-13T11:55:01.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2faa7fc640 1 -- 172.21.15.100:0/1900901656 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 626700363 0 0) 0x7f2fa4001c40 con 0x7f2fac13c9f0 2024-09-13T11:55:01.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2fb13f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:55:01.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.805+0000 7f2faa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:01.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faa7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-13T11:55:01.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faa7fc640 10 monclient: dump: 2024-09-13T11:55:01.826 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:01.826 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:01.826 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.826 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.827 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:01.827 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:01.827 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:01.827 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:01.827 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:01.827 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:01.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faa7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] but i'm connected to v1:172.21.15.100:6789/0, reconnecting 2024-09-13T11:55:01.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faa7fc640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:01.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faa7fc640 1 -- 172.21.15.100:0/1900901656 >> v1:172.21.15.100:6789/0 conn(0x7f2fac13c9f0 legacy=0x7f2fac13ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:01.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faa7fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:55:01.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faa7fc640 1 --2- 172.21.15.100:0/1900901656 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f98003a50 0x7f2f98003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:01.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faa7fc640 10 monclient(hunting): picked mon.b con 0x7f2f98003a50 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:55:01.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faa7fc640 1 --2- 172.21.15.100:0/1900901656 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f98004800 0x7f2f98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:01.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faa7fc640 10 monclient(hunting): picked mon.c con 0x7f2f98004800 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:55:01.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faa7fc640 1 --2- 172.21.15.100:0/1900901656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f980053c0 0x7f2f9800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:01.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faa7fc640 10 monclient(hunting): picked mon.a con 0x7f2f980053c0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:55:01.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faa7fc640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:01.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faa7fc640 1 -- 172.21.15.100:0/1900901656 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f2fac0a2cb0 con 0x7f2f980053c0 2024-09-13T11:55:01.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faa7fc640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:01.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faa7fc640 1 -- 172.21.15.100:0/1900901656 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9800c230 con 0x7f2f98003a50 2024-09-13T11:55:01.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faa7fc640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:01.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faa7fc640 1 -- 172.21.15.100:0/1900901656 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f2f9800c430 con 0x7f2f98004800 2024-09-13T11:55:01.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faa7fc640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:01.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faa7fc640 10 monclient(hunting): _finish_auth 0 2024-09-13T11:55:01.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fb13f8640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-13T11:55:01.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fab7fe640 1 --2- 172.21.15.100:0/1900901656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f980053c0 0x7f2f9800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:01.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fa3fff640 1 --2- 172.21.15.100:0/1900901656 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f98004800 0x7f2f98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:01.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fab7fe640 10 monclient(hunting): get_auth_request con 0x7f2f980053c0 auth_method 0 2024-09-13T11:55:01.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:01.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:01.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fab7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:55:01.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fa3fff640 10 monclient(hunting): get_auth_request con 0x7f2f98004800 auth_method 0 2024-09-13T11:55:01.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fa3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:01.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fa3fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:01.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fa3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:55:01.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faaffd640 1 --2- 172.21.15.100:0/1900901656 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f98003a50 0x7f2f98003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:01.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faaffd640 10 monclient(hunting): get_auth_request con 0x7f2f98003a50 auth_method 0 2024-09-13T11:55:01.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:01.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faaffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:01.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:55:01.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:01.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:01.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:55:01.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fa3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:01.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fa3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:01.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fa3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:55:01.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:01.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:01.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2faaffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:55:01.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fab7fe640 10 monclient(hunting): handle_auth_done global_id 14160 payload 293 2024-09-13T11:55:01.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fab7fe640 1 -- 172.21.15.100:0/1900901656 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f98004800 msgr2=0x7f2f98004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:55:01.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fab7fe640 1 --2- 172.21.15.100:0/1900901656 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f98004800 0x7f2f98004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:01.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fab7fe640 1 -- 172.21.15.100:0/1900901656 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f98003a50 msgr2=0x7f2f98003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:55:01.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fab7fe640 1 --2- 172.21.15.100:0/1900901656 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f98003a50 0x7f2f98003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:01.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fab7fe640 10 monclient: _finish_hunting 0 2024-09-13T11:55:01.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fab7fe640 1 monclient: found mon.a 2024-09-13T11:55:01.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:01.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fab7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:01.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fab7fe640 1 -- 172.21.15.100:0/1900901656 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f98003520 con 0x7f2f980053c0 2024-09-13T11:55:01.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fab7fe640 10 monclient: _finish_auth 0 2024-09-13T11:55:01.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fab7fe640 10 monclient: _check_auth_tickets 2024-09-13T11:55:01.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:01.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.806+0000 7f2fa3fff640 1 --2- 172.21.15.100:0/1900901656 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f98004800 0x7f2f98004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:55:01.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.807+0000 7f2fab7fe640 1 -- 172.21.15.100:0/1900901656 reap_dead start 2024-09-13T11:55:01.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.807+0000 7f2fab7fe640 1 --2- 172.21.15.100:0/1900901656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f980053c0 0x7f2f9800b9c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2fa400aa50 tx=0x7f2f98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368574198cfd5718 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:01.835 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.807+0000 7f2faa7fc640 1 -- 172.21.15.100:0/1900901656 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f94009690 con 0x7f2f980053c0 2024-09-13T11:55:01.835 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.807+0000 7f2faa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:01.835 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.807+0000 7f2faa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:01.835 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.807+0000 7f2faa7fc640 10 monclient: dump: 2024-09-13T11:55:01.835 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:01.835 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:01.835 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.835 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.835 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:01.835 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:01.835 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:01.835 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:01.836 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:01.836 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:01.836 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.807+0000 7f2faa7fc640 1 -- 172.21.15.100:0/1900901656 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fa402f070 con 0x7f2f980053c0 2024-09-13T11:55:01.836 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.807+0000 7f2fb13f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:55:01.836 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.807+0000 7f2faa7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:55:01.836 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.807+0000 7f2fb13f8640 10 monclient: get_monmap_and_config success 2024-09-13T11:55:01.836 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.807+0000 7f2fb13f8640 4 set_mon_vals no callback set 2024-09-13T11:55:01.836 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.807+0000 7f2faa7fc640 1 -- 172.21.15.100:0/1900901656 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa400a670 con 0x7f2f980053c0 2024-09-13T11:55:01.836 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.807+0000 7f2faa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:01.836 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.807+0000 7f2faa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:01.836 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.807+0000 7f2faa7fc640 10 monclient: dump: 2024-09-13T11:55:01.837 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:01.837 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:01.837 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.837 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.837 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:01.837 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:01.837 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:01.837 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:01.837 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:01.837 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:01.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.807+0000 7f2fb13f8640 10 monclient: shutdown 2024-09-13T11:55:01.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.807+0000 7f2fb13f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:55:01.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.807+0000 7f2fb13f8640 1 -- 172.21.15.100:0/1900901656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f980053c0 msgr2=0x7f2f9800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:01.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.807+0000 7f2fb13f8640 1 --2- 172.21.15.100:0/1900901656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f980053c0 0x7f2f9800b9c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2fa400aa50 tx=0x7f2f98005830 comp rx=0 tx=0).stop 2024-09-13T11:55:01.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.807+0000 7f2fb13f8640 1 -- 172.21.15.100:0/1900901656 shutdown_connections 2024-09-13T11:55:01.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.807+0000 7f2fb13f8640 1 --2- 172.21.15.100:0/1900901656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f980053c0 0x7f2f9800b9c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:01.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.807+0000 7f2fb13f8640 1 -- 172.21.15.100:0/1900901656 >> 172.21.15.100:0/1900901656 conn(0x7f2fac0a68e0 msgr2=0x7f2fac0a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:55:01.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.807+0000 7f2fb13f8640 1 -- 172.21.15.100:0/1900901656 shutdown_connections 2024-09-13T11:55:01.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.807+0000 7f2fb13f8640 1 -- 172.21.15.100:0/1900901656 wait complete. 2024-09-13T11:55:01.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.808+0000 7f2fb13f8640 10 monclient: build_initial_monmap 2024-09-13T11:55:01.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.808+0000 7f2fb13f8640 10 monclient: monmap: 2024-09-13T11:55:01.838 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:55:01.839 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:55:01.839 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:55:01.839 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:55:01.839 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:55:01.839 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:01.839 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:55:01.839 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:55:01.839 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:55:01.839 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:01.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.808+0000 7f2fb13f8640 1 Processor -- start 2024-09-13T11:55:01.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.808+0000 7f2fb13f8640 1 -- start start 2024-09-13T11:55:01.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.808+0000 7f2fb13f8640 10 monclient: init 2024-09-13T11:55:01.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fb13f8640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:01.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fb13f8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:55:01.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fb13f8640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f980053c0 0x7f2fac09c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:01.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fb13f8640 10 monclient(hunting): picked mon.noname-a con 0x7f2f980053c0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:55:01.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fb13f8640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fac09c9d0 0x7f2fac09ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:01.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fb13f8640 10 monclient(hunting): picked mon.noname-c con 0x7f2fac09c9d0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:55:01.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fb13f8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2fac1ccb90 0x7f2fac1cd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:01.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fb13f8640 10 monclient(hunting): picked mon.noname-b con 0x7f2fac1ccb90 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:55:01.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fb13f8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:01.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fb13f8640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f2fac0a5be0 con 0x7f2f980053c0 2024-09-13T11:55:01.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fb13f8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:01.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fb13f8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2fac0a5a60 con 0x7f2fac1ccb90 2024-09-13T11:55:01.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fb13f8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:01.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fb13f8640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f2fac06ee30 con 0x7f2fac09c9d0 2024-09-13T11:55:01.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fb13f8640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:01.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2faaffd640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f980053c0 0x7f2fac09c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:01.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fb13f8640 10 monclient(hunting): authenticate will time out at 882.402527s 2024-09-13T11:55:01.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2faaffd640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f980053c0 0x7f2fac09c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53466/0 (socket says 172.21.15.100:53466) 2024-09-13T11:55:01.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2faaffd640 1 -- 172.21.15.100:0/4099272239 learned_addr learned my addr 172.21.15.100:0/4099272239 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:55:01.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fa3fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fac09c9d0 0x7f2fac09ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:01.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2faaffd640 10 monclient(hunting): get_auth_request con 0x7f2f980053c0 auth_method 0 2024-09-13T11:55:01.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2faaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:01.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2faaffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:01.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2faaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:01.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fa3fff640 10 monclient(hunting): get_auth_request con 0x7f2fac09c9d0 auth_method 0 2024-09-13T11:55:01.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fa3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:01.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fa3fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:01.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:01.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2faaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:01.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2faaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:01.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2faaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:01.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fa3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:01.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fa3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:01.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fa3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:01.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fab7fe640 1 --2- 172.21.15.100:0/4099272239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2fac1ccb90 0x7f2fac1cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:01.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fab7fe640 10 monclient(hunting): get_auth_request con 0x7f2fac1ccb90 auth_method 0 2024-09-13T11:55:01.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:01.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:01.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:01.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:01.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:01.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:01.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2faaffd640 10 monclient(hunting): handle_auth_done global_id 14163 payload 824 2024-09-13T11:55:01.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2faaffd640 1 -- 172.21.15.100:0/4099272239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fac09c9d0 msgr2=0x7f2fac09ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:01.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2faaffd640 1 --2- 172.21.15.100:0/4099272239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fac09c9d0 0x7f2fac09ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:01.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2faaffd640 1 -- 172.21.15.100:0/4099272239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2fac1ccb90 msgr2=0x7f2fac1cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:01.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2faaffd640 1 --2- 172.21.15.100:0/4099272239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2fac1ccb90 0x7f2fac1cd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:01.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2faaffd640 10 monclient: _finish_hunting 0 2024-09-13T11:55:01.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2faaffd640 1 monclient: found mon.noname-a 2024-09-13T11:55:01.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2faaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:01.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2faaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:55:01.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2faaffd640 1 -- 172.21.15.100:0/4099272239 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fac09f990 con 0x7f2f980053c0 2024-09-13T11:55:01.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.809+0000 7f2fa3fff640 1 --2- 172.21.15.100:0/4099272239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fac09c9d0 0x7f2fac09ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:55:01.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.810+0000 7f2faaffd640 1 --2- 172.21.15.100:0/4099272239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f980053c0 0x7f2fac09c490 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c00f820 tx=0x7f2f9c00fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:01.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.810+0000 7f2fa37fe640 1 -- 172.21.15.100:0/4099272239 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c01a070 con 0x7f2f980053c0 2024-09-13T11:55:01.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.810+0000 7f2fa37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:01.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.810+0000 7f2fa37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:55:01.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.810+0000 7f2fa37fe640 10 monclient: dump: 2024-09-13T11:55:01.845 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:01.845 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:01.845 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.845 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.845 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:01.845 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:01.845 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:01.845 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:01.845 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:01.846 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:01.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.810+0000 7f2fa37fe640 10 monclient: _finish_auth 0 2024-09-13T11:55:01.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.810+0000 7f2fa37fe640 10 monclient: _check_auth_tickets 2024-09-13T11:55:01.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.810+0000 7f2fa37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:01.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.810+0000 7f2fb13f8640 5 monclient: authenticate success, global_id 14163 2024-09-13T11:55:01.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.810+0000 7f2fa37fe640 1 -- 172.21.15.100:0/4099272239 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f9c005110 con 0x7f2f980053c0 2024-09-13T11:55:01.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.810+0000 7f2fa37fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:55:01.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.810+0000 7f2fa37fe640 1 -- 172.21.15.100:0/4099272239 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c007050 con 0x7f2f980053c0 2024-09-13T11:55:01.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.810+0000 7f2fb13f8640 10 monclient: _renew_subs 2024-09-13T11:55:01.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.810+0000 7f2fb13f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:01.846 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.810+0000 7f2fb13f8640 1 -- 172.21.15.100:0/4099272239 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fac09fc70 con 0x7f2f980053c0 2024-09-13T11:55:01.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.810+0000 7f2fa9ffb640 4 set_mon_vals no callback set 2024-09-13T11:55:01.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.810+0000 7f2fa37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:01.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.810+0000 7f2fa37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:01.847 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.810+0000 7f2fa37fe640 10 monclient: dump: 2024-09-13T11:55:01.847 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:01.847 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:01.847 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.847 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.847 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:01.847 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:01.847 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:01.847 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:01.848 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:01.848 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:01.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.810+0000 7f2fb13f8640 10 monclient: _renew_subs 2024-09-13T11:55:01.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.810+0000 7f2fb13f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:01.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.810+0000 7f2fb13f8640 1 -- 172.21.15.100:0/4099272239 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fac0a10d0 con 0x7f2f980053c0 2024-09-13T11:55:01.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.810+0000 7f2fb13f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:01.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.810+0000 7f2fb13f8640 1 --2- 172.21.15.100:0/4099272239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fac1cd550 0x7f2fac139050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:01.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.811+0000 7f2fb13f8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:01.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.811+0000 7f2fb13f8640 1 -- 172.21.15.100:0/4099272239 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f2fac06f210 con 0x7f2fac1cd550 2024-09-13T11:55:01.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.811+0000 7f2fa3fff640 1 --2- 172.21.15.100:0/4099272239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fac1cd550 0x7f2fac139050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:01.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.811+0000 7f2fa3fff640 10 monclient: get_auth_request con 0x7f2fac1cd550 auth_method 0 2024-09-13T11:55:01.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.811+0000 7f2fa3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:01.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.811+0000 7f2fa3fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:01.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.811+0000 7f2fa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:01.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.811+0000 7f2fa3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:01.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.811+0000 7f2fa3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:01.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.811+0000 7f2fa3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:01.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.811+0000 7f2fa37fe640 1 -- 172.21.15.100:0/4099272239 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f2f9c01d070 con 0x7f2f980053c0 2024-09-13T11:55:01.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.811+0000 7f2fa3fff640 10 monclient(hunting): handle_auth_done global_id 14166 payload 824 2024-09-13T11:55:01.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.811+0000 7f2fa3fff640 1 -- 172.21.15.100:0/4099272239 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2fac09f3c0 con 0x7f2fac1cd550 2024-09-13T11:55:01.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.811+0000 7f2fa3fff640 1 --2- 172.21.15.100:0/4099272239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fac1cd550 0x7f2fac139050 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2f94014de0 tx=0x7f2f94012700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:01.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.811+0000 7f2fa37fe640 1 --2- 172.21.15.100:0/4099272239 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f2f8c03dfa0 0x7f2f8c040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:01.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.811+0000 7f2fa37fe640 1 -- 172.21.15.100:0/4099272239 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4346+0+0 (secure 0 0 0) 0x7f2f9c051ca0 con 0x7f2f980053c0 2024-09-13T11:55:01.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.811+0000 7f2fa37fe640 1 -- 172.21.15.100:0/4099272239 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f94004030 con 0x7f2fac1cd550 2024-09-13T11:55:01.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.811+0000 7f2fa37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:01.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.811+0000 7f2fa37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:01.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.811+0000 7f2fa37fe640 10 monclient: dump: 2024-09-13T11:55:01.850 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:01.850 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:01.850 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.850 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.850 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:01.850 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:01.850 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:01.850 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:01.851 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:01.851 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:01.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.811+0000 7f2fab7fe640 1 --2- 172.21.15.100:0/4099272239 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f2f8c03dfa0 0x7f2f8c040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:01.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.811+0000 7f2fab7fe640 10 monclient: get_auth_request con 0x7f2f8c03dfa0 auth_method 0 2024-09-13T11:55:01.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.812+0000 7f2fa37fe640 1 -- 172.21.15.100:0/4099272239 <== mon.0 v2:172.21.15.100:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2f94004030 con 0x7f2fac1cd550 2024-09-13T11:55:01.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.812+0000 7f2fa37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:01.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.812+0000 7f2fa37fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:55:01.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.812+0000 7f2fa37fe640 1 -- 172.21.15.100:0/4099272239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fac1cd550 msgr2=0x7f2fac139050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:01.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.812+0000 7f2fa37fe640 1 --2- 172.21.15.100:0/4099272239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fac1cd550 0x7f2fac139050 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2f94014de0 tx=0x7f2f94012700 comp rx=0 tx=0).stop 2024-09-13T11:55:01.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.812+0000 7f2fa37fe640 1 -- 172.21.15.100:0/4099272239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fac1cd550 msgr2=0x7f2fac139050 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:55:01.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.812+0000 7f2fa37fe640 1 --2- 172.21.15.100:0/4099272239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fac1cd550 0x7f2fac139050 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2f94014de0 tx=0x7f2f94012700 comp rx=0 tx=0).stop 2024-09-13T11:55:01.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.812+0000 7f2fab7fe640 1 --2- 172.21.15.100:0/4099272239 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f2f8c03dfa0 0x7f2f8c040460 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2f9800c200 tx=0x7f2fa4007000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:01.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fb13f8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:55:01.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fb13f8640 1 --2- 172.21.15.100:0/4099272239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fac139590 0x7f2fac1399a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:01.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fb13f8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:01.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fb13f8640 1 -- 172.21.15.100:0/4099272239 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f2fac09ded0 con 0x7f2fac139590 2024-09-13T11:55:01.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fa3fff640 1 --2- 172.21.15.100:0/4099272239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fac139590 0x7f2fac1399a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:01.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fa3fff640 10 monclient: get_auth_request con 0x7f2fac139590 auth_method 0 2024-09-13T11:55:01.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fa3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:01.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fa3fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:01.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:01.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fa3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:01.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fa3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:01.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fa3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:01.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fa3fff640 10 monclient(hunting): handle_auth_done global_id 14169 payload 824 2024-09-13T11:55:01.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fa3fff640 1 -- 172.21.15.100:0/4099272239 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2fac1cd930 con 0x7f2fac139590 2024-09-13T11:55:01.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fa3fff640 1 --2- 172.21.15.100:0/4099272239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fac139590 0x7f2fac1399a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2f94012c50 tx=0x7f2f94004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:01.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fa37fe640 1 -- 172.21.15.100:0/4099272239 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f94005c60 con 0x7f2fac139590 2024-09-13T11:55:01.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fa37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:01.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fa37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:01.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fa37fe640 10 monclient: dump: 2024-09-13T11:55:01.854 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:01.854 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:01.854 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.854 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.854 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:01.854 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:01.854 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:01.854 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:01.854 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:01.854 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:01.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fa37fe640 1 -- 172.21.15.100:0/4099272239 <== mon.0 v2:172.21.15.100:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f2f94005c60 con 0x7f2fac139590 2024-09-13T11:55:01.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fa37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:55:01.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fa37fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:55:01.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fa37fe640 1 -- 172.21.15.100:0/4099272239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fac139590 msgr2=0x7f2fac1399a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:01.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fa37fe640 1 --2- 172.21.15.100:0/4099272239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fac139590 0x7f2fac1399a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2f94012c50 tx=0x7f2f94004730 comp rx=0 tx=0).stop 2024-09-13T11:55:01.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fa37fe640 1 -- 172.21.15.100:0/4099272239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fac139590 msgr2=0x7f2fac1399a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:55:01.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.818+0000 7f2fa37fe640 1 --2- 172.21.15.100:0/4099272239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fac139590 0x7f2fac1399a0 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2f94012c50 tx=0x7f2f94004730 comp rx=0 tx=0).stop 2024-09-13T11:55:01.855 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:55:01.856 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:55:01.856 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:55:01.856 INFO:teuthology.orchestra.run.smithi100.stdout: "state": "leader", 2024-09-13T11:55:01.856 INFO:teuthology.orchestra.run.smithi100.stdout: "election_epoch": 16, 2024-09-13T11:55:01.856 INFO:teuthology.orchestra.run.smithi100.stdout: "uptime": 2817, 2024-09-13T11:55:01.856 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum": [ 2024-09-13T11:55:01.856 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-13T11:55:01.856 INFO:teuthology.orchestra.run.smithi100.stdout: 1, 2024-09-13T11:55:01.856 INFO:teuthology.orchestra.run.smithi100.stdout: 2 2024-09-13T11:55:01.856 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:01.856 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_age": 2, 2024-09-13T11:55:01.857 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:55:01.857 INFO:teuthology.orchestra.run.smithi100.stdout: "required_con": "2450521708007866372", 2024-09-13T11:55:01.857 INFO:teuthology.orchestra.run.smithi100.stdout: "required_mon": [ 2024-09-13T11:55:01.857 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:55:01.857 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:55:01.857 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:55:01.857 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:55:01.857 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:55:01.857 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:55:01.857 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:55:01.857 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:55:01.857 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:55:01.858 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:55:01.858 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:55:01.858 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:01.858 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:55:01.858 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_mon": [ 2024-09-13T11:55:01.858 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:55:01.858 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:55:01.858 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:55:01.858 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:55:01.858 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:55:01.858 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:55:01.858 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:55:01.859 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:55:01.859 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:55:01.859 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:55:01.859 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:55:01.859 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:01.859 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:01.859 INFO:teuthology.orchestra.run.smithi100.stdout: "outside_quorum": [], 2024-09-13T11:55:01.859 INFO:teuthology.orchestra.run.smithi100.stdout: "extra_probe_peers": [], 2024-09-13T11:55:01.859 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_provider": [], 2024-09-13T11:55:01.859 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": { 2024-09-13T11:55:01.860 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-09-13T11:55:01.860 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "0b1f4b4d-fad4-4dd3-86ac-a286d80846c0", 2024-09-13T11:55:01.860 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:55:01.860 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:55:01.860 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release": 19, 2024-09-13T11:55:01.860 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release_name": "squid", 2024-09-13T11:55:01.860 INFO:teuthology.orchestra.run.smithi100.stdout: "election_strategy": 1, 2024-09-13T11:55:01.860 INFO:teuthology.orchestra.run.smithi100.stdout: "disallowed_leaders: ": "", 2024-09-13T11:55:01.860 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false, 2024-09-13T11:55:01.860 INFO:teuthology.orchestra.run.smithi100.stdout: "tiebreaker_mon": "", 2024-09-13T11:55:01.860 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_ranks: ": "", 2024-09-13T11:55:01.860 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:55:01.861 INFO:teuthology.orchestra.run.smithi100.stdout: "persistent": [ 2024-09-13T11:55:01.861 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:55:01.861 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:55:01.861 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:55:01.861 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:55:01.861 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:55:01.861 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:55:01.861 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:55:01.861 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:55:01.861 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:55:01.861 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:55:01.862 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:55:01.862 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:01.862 INFO:teuthology.orchestra.run.smithi100.stdout: "optional": [] 2024-09-13T11:55:01.862 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:01.862 INFO:teuthology.orchestra.run.smithi100.stdout: "mons": [ 2024-09-13T11:55:01.862 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:01.862 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:55:01.862 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:55:01.862 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:55:01.862 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:55:01.862 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:01.862 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:55:01.862 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3300", 2024-09-13T11:55:01.863 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:01.863 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:01.863 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:01.863 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:55:01.863 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789", 2024-09-13T11:55:01.863 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:01.863 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:01.863 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:01.863 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:01.863 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789/0", 2024-09-13T11:55:01.863 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-13T11:55:01.863 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:55:01.863 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:55:01.864 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:55:01.864 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:01.864 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:01.864 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:55:01.864 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:55:01.864 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:55:01.864 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:55:01.864 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:01.864 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:55:01.864 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:3300", 2024-09-13T11:55:01.864 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:01.864 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:01.864 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:01.865 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:55:01.865 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789", 2024-09-13T11:55:01.865 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:01.865 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:01.865 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:01.865 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:01.865 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789/0", 2024-09-13T11:55:01.865 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.122:6789/0", 2024-09-13T11:55:01.865 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:55:01.865 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:55:01.865 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:55:01.865 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:01.865 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:01.866 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:55:01.866 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:55:01.866 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:55:01.866 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:55:01.866 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:01.866 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:55:01.866 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3301", 2024-09-13T11:55:01.866 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:01.866 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:01.866 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:01.866 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:55:01.866 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790", 2024-09-13T11:55:01.866 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:01.866 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:01.867 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:01.867 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:01.867 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790/0", 2024-09-13T11:55:01.867 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-13T11:55:01.867 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:55:01.867 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:55:01.867 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:55:01.867 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:01.867 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:01.867 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:01.867 INFO:teuthology.orchestra.run.smithi100.stdout: "feature_map": { 2024-09-13T11:55:01.867 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": [ 2024-09-13T11:55:01.867 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:01.868 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:55:01.868 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:55:01.868 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:55:01.868 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:01.868 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:01.868 INFO:teuthology.orchestra.run.smithi100.stdout: "client": [ 2024-09-13T11:55:01.868 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:01.868 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:55:01.868 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:55:01.868 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 3 2024-09-13T11:55:01.868 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:01.868 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:01.868 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:01.869 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false 2024-09-13T11:55:01.869 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:55:01.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.819+0000 7f2fb13f8640 1 -- 172.21.15.100:0/4099272239 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f2f8c03dfa0 msgr2=0x7f2f8c040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:01.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.819+0000 7f2fb13f8640 1 --2- 172.21.15.100:0/4099272239 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f2f8c03dfa0 0x7f2f8c040460 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2f9800c200 tx=0x7f2fa4007000 comp rx=0 tx=0).stop 2024-09-13T11:55:01.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.819+0000 7f2fb13f8640 10 monclient: shutdown 2024-09-13T11:55:01.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.819+0000 7f2fb13f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:55:01.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.819+0000 7f2fb13f8640 1 -- 172.21.15.100:0/4099272239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f980053c0 msgr2=0x7f2fac09c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:01.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.819+0000 7f2fb13f8640 1 --2- 172.21.15.100:0/4099272239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f980053c0 0x7f2fac09c490 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c00f820 tx=0x7f2f9c00fc20 comp rx=0 tx=0).stop 2024-09-13T11:55:01.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.819+0000 7f2fab7fe640 1 -- 172.21.15.100:0/4099272239 reap_dead start 2024-09-13T11:55:01.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.819+0000 7f2fab7fe640 1 -- 172.21.15.100:0/4099272239 reap_dead start 2024-09-13T11:55:01.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.819+0000 7f2fb13f8640 1 -- 172.21.15.100:0/4099272239 shutdown_connections 2024-09-13T11:55:01.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.819+0000 7f2fb13f8640 1 -- 172.21.15.100:0/4099272239 >> 172.21.15.100:0/4099272239 conn(0x7f2fac0a68e0 msgr2=0x7f2fac0a7c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:55:01.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.819+0000 7f2fb13f8640 1 -- 172.21.15.100:0/4099272239 shutdown_connections 2024-09-13T11:55:01.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.819+0000 7f2fb13f8640 1 -- 172.21.15.100:0/4099272239 wait complete. 2024-09-13T11:55:01.871 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T11:55:01.871 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-13T11:55:01.871 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T11:55:01.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.967+0000 7f8d77223640 10 monclient: get_monmap_and_config 2024-09-13T11:55:01.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.967+0000 7f8d77223640 10 monclient: build_initial_monmap 2024-09-13T11:55:01.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.967+0000 7f8d77223640 10 monclient: monmap: 2024-09-13T11:55:01.969 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:55:01.969 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:01.969 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:55:01.968514+0000 2024-09-13T11:55:01.969 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:55:01.968514+0000 2024-09-13T11:55:01.970 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:55:01.970 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:01.970 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:55:01.970 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:55:01.970 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:55:01.970 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:55:01.970 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:55:01.970 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:01.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.968+0000 7f8d77223640 1 Processor -- start 2024-09-13T11:55:01.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.968+0000 7f8d77223640 1 -- start start 2024-09-13T11:55:01.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.968+0000 7f8d77223640 10 monclient: init 2024-09-13T11:55:01.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.968+0000 7f8d77223640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:01.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.968+0000 7f8d77223640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-13T11:55:01.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.968+0000 7f8d77223640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8d70073550 addr v1:172.21.15.100:6789/0 2024-09-13T11:55:01.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.968+0000 7f8d77223640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8d700a1fe0 0x7f8d7013a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:01.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.968+0000 7f8d77223640 10 monclient(hunting): picked mon.noname-a con 0x7f8d700a1fe0 addr v2:172.21.15.100:3300/0 2024-09-13T11:55:01.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.968+0000 7f8d77223640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8d7013aaf0 addr v1:172.21.15.122:6789/0 2024-09-13T11:55:01.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.968+0000 7f8d77223640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d700a0db0 con 0x7f8d70073550 2024-09-13T11:55:01.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.968+0000 7f8d77223640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d700a1020 con 0x7f8d7013aaf0 2024-09-13T11:55:01.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.968+0000 7f8d77223640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:01.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.968+0000 7f8d77223640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f8d7009fda0 con 0x7f8d700a1fe0 2024-09-13T11:55:01.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.968+0000 7f8d77223640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:01.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.968+0000 7f8d77223640 10 monclient(hunting): authenticate will time out at 882.561978s 2024-09-13T11:55:01.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.969+0000 7f8d6ffff640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8d700a1fe0 0x7f8d7013a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:01.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.969+0000 7f8d74fbf640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f8d70073550 0x7f8d70073930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:48274/0 (socket says 172.21.15.100:48274) 2024-09-13T11:55:01.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.969+0000 7f8d6ffff640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8d700a1fe0 0x7f8d7013a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53480/0 (socket says 172.21.15.100:53480) 2024-09-13T11:55:01.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.969+0000 7f8d74fbf640 1 -- 172.21.15.100:0/2613622031 learned_addr learned my addr 172.21.15.100:0/2613622031 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:55:01.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.969+0000 7f8d6ffff640 10 monclient(hunting): get_auth_request con 0x7f8d700a1fe0 auth_method 0 2024-09-13T11:55:01.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.969+0000 7f8d6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:01.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.969+0000 7f8d6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:01.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.969+0000 7f8d6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:01.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.969+0000 7f8d6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:01.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.969+0000 7f8d6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:01.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.969+0000 7f8d6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:01.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.969+0000 7f8d6ffff640 10 monclient(hunting): handle_auth_done global_id 14172 payload 274 2024-09-13T11:55:01.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.969+0000 7f8d6ffff640 1 -- 172.21.15.100:0/2613622031 >> v1:172.21.15.122:6789/0 conn(0x7f8d7013aaf0 legacy=0x7f8d7013cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:01.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.969+0000 7f8d6ffff640 1 -- 172.21.15.100:0/2613622031 >> v1:172.21.15.100:6789/0 conn(0x7f8d70073550 legacy=0x7f8d70073930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:01.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.969+0000 7f8d6ffff640 10 monclient: _finish_hunting 0 2024-09-13T11:55:01.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.969+0000 7f8d6ffff640 1 monclient: found mon.noname-a 2024-09-13T11:55:01.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.969+0000 7f8d6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:01.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.969+0000 7f8d6ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:55:01.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.969+0000 7f8d6ffff640 1 -- 172.21.15.100:0/2613622031 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d700a1300 con 0x7f8d700a1fe0 2024-09-13T11:55:01.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.969+0000 7f8d6f7fe640 1 -- 172.21.15.100:0/2613622031 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 648170484 0 0) 0x7f8d700a0db0 con 0x7f8d70073550 2024-09-13T11:55:01.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.969+0000 7f8d6f7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:55:01.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d6ffff640 1 --2- 172.21.15.100:0/2613622031 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d700a1fe0 0x7f8d7013a5b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0097a0 tx=0x7f8d5c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e58d9b0a5148ac7d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:01.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d6f7fe640 1 -- 172.21.15.100:0/2613622031 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c037070 con 0x7f8d700a1fe0 2024-09-13T11:55:01.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:01.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d6f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:55:01.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d6f7fe640 10 monclient: dump: 2024-09-13T11:55:01.985 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:01.985 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:01.985 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.985 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.985 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:01.986 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:01.986 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:01.986 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:01.986 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:01.986 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:01.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d6f7fe640 10 monclient: _finish_auth 0 2024-09-13T11:55:01.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d6f7fe640 10 monclient: _check_auth_tickets 2024-09-13T11:55:01.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:01.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d6f7fe640 1 -- 172.21.15.100:0/2613622031 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d5c032440 con 0x7f8d700a1fe0 2024-09-13T11:55:01.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d77223640 5 monclient: authenticate success, global_id 14172 2024-09-13T11:55:01.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d77223640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:55:01.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d6f7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:55:01.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d6f7fe640 1 -- 172.21.15.100:0/2613622031 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c02ae60 con 0x7f8d700a1fe0 2024-09-13T11:55:01.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:01.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d6f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:01.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d6f7fe640 10 monclient: dump: 2024-09-13T11:55:01.987 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:01.987 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:01.987 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.987 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.987 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:01.987 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:01.987 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:01.988 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:01.988 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:01.988 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:01.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d77223640 10 monclient: get_monmap_and_config success 2024-09-13T11:55:01.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d77223640 4 set_mon_vals no callback set 2024-09-13T11:55:01.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d77223640 10 monclient: shutdown 2024-09-13T11:55:01.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d77223640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:55:01.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d77223640 1 -- 172.21.15.100:0/2613622031 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d700a1fe0 msgr2=0x7f8d7013a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:01.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d77223640 1 --2- 172.21.15.100:0/2613622031 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d700a1fe0 0x7f8d7013a5b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0097a0 tx=0x7f8d5c02a110 comp rx=0 tx=0).stop 2024-09-13T11:55:01.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d77223640 1 -- 172.21.15.100:0/2613622031 shutdown_connections 2024-09-13T11:55:01.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d77223640 1 --2- 172.21.15.100:0/2613622031 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d700a1fe0 0x7f8d7013a5b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:01.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d77223640 1 -- 172.21.15.100:0/2613622031 >> 172.21.15.100:0/2613622031 conn(0x7f8d700a83f0 msgr2=0x7f8d700a48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:55:01.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d77223640 1 -- 172.21.15.100:0/2613622031 shutdown_connections 2024-09-13T11:55:01.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.970+0000 7f8d77223640 1 -- 172.21.15.100:0/2613622031 wait complete. 2024-09-13T11:55:01.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.971+0000 7f8d77223640 10 monclient: build_initial_monmap 2024-09-13T11:55:01.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.971+0000 7f8d77223640 10 monclient: monmap: 2024-09-13T11:55:01.989 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:55:01.989 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:55:01.989 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:55:01.989 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:55:01.989 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:55:01.989 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:01.989 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:55:01.990 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:55:01.990 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:55:01.990 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:01.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.971+0000 7f8d77223640 1 Processor -- start 2024-09-13T11:55:01.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.971+0000 7f8d77223640 1 -- start start 2024-09-13T11:55:01.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.971+0000 7f8d77223640 10 monclient: init 2024-09-13T11:55:01.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d77223640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:01.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d77223640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:55:01.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d77223640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d70073550 0x7f8d70139bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:01.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d77223640 10 monclient(hunting): picked mon.noname-c con 0x7f8d70073550 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:55:01.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d77223640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d700a1fe0 0x7f8d701cac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:01.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d77223640 10 monclient(hunting): picked mon.noname-b con 0x7f8d700a1fe0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:55:01.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d77223640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d7013aaf0 0x7f8d701375a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:01.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d77223640 10 monclient(hunting): picked mon.noname-a con 0x7f8d7013aaf0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:55:01.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d77223640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:01.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d77223640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f8d70070f40 con 0x7f8d7013aaf0 2024-09-13T11:55:01.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d77223640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:01.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d77223640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f8d700a16a0 con 0x7f8d700a1fe0 2024-09-13T11:55:01.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d77223640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:01.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d77223640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f8d700710c0 con 0x7f8d70073550 2024-09-13T11:55:01.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d77223640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:01.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d77223640 10 monclient(hunting): authenticate will time out at 882.565426s 2024-09-13T11:55:01.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d74fbf640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d70073550 0x7f8d70139bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:01.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d757c0640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d7013aaf0 0x7f8d701375a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:01.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d74fbf640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d70073550 0x7f8d70139bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:36952/0 (socket says 172.21.15.100:36952) 2024-09-13T11:55:01.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d74fbf640 1 -- 172.21.15.100:0/2687184121 learned_addr learned my addr 172.21.15.100:0/2687184121 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:55:01.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d757c0640 10 monclient(hunting): get_auth_request con 0x7f8d7013aaf0 auth_method 0 2024-09-13T11:55:01.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d757c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:01.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d757c0640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:01.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d757c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:01.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d74fbf640 10 monclient(hunting): get_auth_request con 0x7f8d70073550 auth_method 0 2024-09-13T11:55:01.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d74fbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:01.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d74fbf640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:01.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d74fbf640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:01.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d757c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:01.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d757c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:01.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d757c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:01.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d74fbf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:01.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d74fbf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:01.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d6ffff640 1 --2- 172.21.15.100:0/2687184121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d700a1fe0 0x7f8d701cac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:01.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d74fbf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:01.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d6ffff640 10 monclient(hunting): get_auth_request con 0x7f8d700a1fe0 auth_method 0 2024-09-13T11:55:01.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:01.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:01.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:01.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:01.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:01.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:01.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d757c0640 10 monclient(hunting): handle_auth_done global_id 14178 payload 824 2024-09-13T11:55:01.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d757c0640 1 -- 172.21.15.100:0/2687184121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d70073550 msgr2=0x7f8d70139bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:01.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d757c0640 1 --2- 172.21.15.100:0/2687184121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d70073550 0x7f8d70139bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:01.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d757c0640 1 -- 172.21.15.100:0/2687184121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d700a1fe0 msgr2=0x7f8d701cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:01.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d757c0640 1 --2- 172.21.15.100:0/2687184121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d700a1fe0 0x7f8d701cac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:01.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d757c0640 10 monclient: _finish_hunting 0 2024-09-13T11:55:01.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d757c0640 1 monclient: found mon.noname-a 2024-09-13T11:55:01.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d757c0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:01.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d757c0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:55:01.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d757c0640 1 -- 172.21.15.100:0/2687184121 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d7013e3d0 con 0x7f8d7013aaf0 2024-09-13T11:55:01.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d74fbf640 1 --2- 172.21.15.100:0/2687184121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d70073550 0x7f8d70139bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:55:01.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.972+0000 7f8d757c0640 1 --2- 172.21.15.100:0/2687184121 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d7013aaf0 0x7f8d701375a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8d70070ef0 tx=0x7f8d6800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:01.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.973+0000 7f8d6d7fa640 1 -- 172.21.15.100:0/2687184121 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d68003b70 con 0x7f8d7013aaf0 2024-09-13T11:55:01.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.973+0000 7f8d6d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:01.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.973+0000 7f8d6d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:55:01.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.973+0000 7f8d6d7fa640 10 monclient: dump: 2024-09-13T11:55:01.995 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:01.995 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:01.996 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.996 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.996 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:01.996 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:01.996 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:01.996 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:01.996 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:01.996 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:01.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.973+0000 7f8d6d7fa640 10 monclient: _finish_auth 0 2024-09-13T11:55:01.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.973+0000 7f8d6d7fa640 10 monclient: _check_auth_tickets 2024-09-13T11:55:01.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.973+0000 7f8d6d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:01.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.973+0000 7f8d6d7fa640 1 -- 172.21.15.100:0/2687184121 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d68003d10 con 0x7f8d7013aaf0 2024-09-13T11:55:01.997 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.973+0000 7f8d77223640 5 monclient: authenticate success, global_id 14178 2024-09-13T11:55:01.997 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.973+0000 7f8d77223640 10 monclient: _renew_subs 2024-09-13T11:55:01.997 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.973+0000 7f8d77223640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:01.997 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.973+0000 7f8d77223640 1 -- 172.21.15.100:0/2687184121 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d7009f0b0 con 0x7f8d7013aaf0 2024-09-13T11:55:01.997 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.973+0000 7f8d6d7fa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:55:01.997 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.973+0000 7f8d6d7fa640 1 -- 172.21.15.100:0/2687184121 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d680052f0 con 0x7f8d7013aaf0 2024-09-13T11:55:01.997 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.973+0000 7f8d6d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:01.997 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.973+0000 7f8d6d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:01.997 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.973+0000 7f8d6d7fa640 10 monclient: dump: 2024-09-13T11:55:01.997 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:01.997 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:01.997 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.998 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:01.998 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:01.998 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:01.998 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:01.998 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:01.998 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:01.998 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:01.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.973+0000 7f8d6effd640 4 set_mon_vals no callback set 2024-09-13T11:55:01.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.973+0000 7f8d77223640 10 monclient: _renew_subs 2024-09-13T11:55:01.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.973+0000 7f8d77223640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:01.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.973+0000 7f8d77223640 1 -- 172.21.15.100:0/2687184121 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d70072ae0 con 0x7f8d7013aaf0 2024-09-13T11:55:01.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d77223640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:01.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d6d7fa640 1 -- 172.21.15.100:0/2687184121 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f8d680056a0 con 0x7f8d7013aaf0 2024-09-13T11:55:01.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d77223640 1 --2- 172.21.15.100:0/2687184121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7011e630 0x7f8d70137ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:01.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d77223640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:01.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d77223640 1 -- 172.21.15.100:0/2687184121 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f8d7011ee20 con 0x7f8d7011e630 2024-09-13T11:55:01.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d74fbf640 1 --2- 172.21.15.100:0/2687184121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7011e630 0x7f8d70137ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:01.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d74fbf640 10 monclient: get_auth_request con 0x7f8d7011e630 auth_method 0 2024-09-13T11:55:01.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d74fbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:01.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d74fbf640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:01.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d74fbf640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:01.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d74fbf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:01.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d74fbf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:02.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d74fbf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:02.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d6d7fa640 1 --2- 172.21.15.100:0/2687184121 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f8d5403dfa0 0x7f8d54040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d6d7fa640 1 -- 172.21.15.100:0/2687184121 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4346+0+0 (secure 0 0 0) 0x7f8d68052550 con 0x7f8d7013aaf0 2024-09-13T11:55:02.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d74fbf640 10 monclient(hunting): handle_auth_done global_id 4877 payload 824 2024-09-13T11:55:02.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d74fbf640 1 -- 172.21.15.100:0/2687184121 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8d70073930 con 0x7f8d7011e630 2024-09-13T11:55:02.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d74fbf640 1 --2- 172.21.15.100:0/2687184121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7011e630 0x7f8d70137ae0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8d64000f60 tx=0x7f8d64001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:02.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d6ffff640 1 --2- 172.21.15.100:0/2687184121 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f8d5403dfa0 0x7f8d54040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:02.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d6ffff640 10 monclient: get_auth_request con 0x7f8d5403dfa0 auth_method 0 2024-09-13T11:55:02.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d6d7fa640 1 -- 172.21.15.100:0/2687184121 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d6400fa20 con 0x7f8d7011e630 2024-09-13T11:55:02.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d6d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:02.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d6d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:55:02.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d6d7fa640 10 monclient: dump: 2024-09-13T11:55:02.001 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:02.001 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:02.001 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.001 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.001 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:02.001 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.001 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:02.001 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:02.001 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:02.001 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d6d7fa640 1 -- 172.21.15.100:0/2687184121 <== mon.2 v2:172.21.15.100:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8d64018070 con 0x7f8d7011e630 2024-09-13T11:55:02.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d6d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:02.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d6d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:55:02.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d6d7fa640 1 -- 172.21.15.100:0/2687184121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7011e630 msgr2=0x7f8d70137ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d6d7fa640 1 --2- 172.21.15.100:0/2687184121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7011e630 0x7f8d70137ae0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8d64000f60 tx=0x7f8d64001e70 comp rx=0 tx=0).stop 2024-09-13T11:55:02.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d6d7fa640 1 -- 172.21.15.100:0/2687184121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7011e630 msgr2=0x7f8d70137ae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:55:02.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.974+0000 7f8d6d7fa640 1 --2- 172.21.15.100:0/2687184121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7011e630 0x7f8d70137ae0 secure :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8d64000f60 tx=0x7f8d64001e70 comp rx=0 tx=0).stop 2024-09-13T11:55:02.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.975+0000 7f8d6ffff640 1 --2- 172.21.15.100:0/2687184121 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f8d5403dfa0 0x7f8d54040460 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c034000 tx=0x7f8d5c02e000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:02.004 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.981+0000 7f8d77223640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:55:02.004 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.981+0000 7f8d77223640 1 --2- 172.21.15.100:0/2687184121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d70138020 0x7f8d7013aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.004 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.981+0000 7f8d77223640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:02.004 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.981+0000 7f8d77223640 1 -- 172.21.15.100:0/2687184121 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f8d70138430 con 0x7f8d70138020 2024-09-13T11:55:02.004 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.981+0000 7f8d74fbf640 1 --2- 172.21.15.100:0/2687184121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d70138020 0x7f8d7013aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:02.004 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.981+0000 7f8d74fbf640 10 monclient: get_auth_request con 0x7f8d70138020 auth_method 0 2024-09-13T11:55:02.004 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.981+0000 7f8d74fbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:02.004 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.981+0000 7f8d74fbf640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:02.004 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.981+0000 7f8d74fbf640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:02.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.981+0000 7f8d74fbf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:02.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.981+0000 7f8d74fbf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:02.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.981+0000 7f8d74fbf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:02.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.981+0000 7f8d74fbf640 10 monclient(hunting): handle_auth_done global_id 4880 payload 824 2024-09-13T11:55:02.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.981+0000 7f8d74fbf640 1 -- 172.21.15.100:0/2687184121 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8d701c34d0 con 0x7f8d70138020 2024-09-13T11:55:02.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.981+0000 7f8d74fbf640 1 --2- 172.21.15.100:0/2687184121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d70138020 0x7f8d7013aed0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8d64000f60 tx=0x7f8d64000ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:02.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.981+0000 7f8d6d7fa640 1 -- 172.21.15.100:0/2687184121 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d64013740 con 0x7f8d70138020 2024-09-13T11:55:02.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.981+0000 7f8d6d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:02.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.981+0000 7f8d6d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:55:02.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.981+0000 7f8d6d7fa640 10 monclient: dump: 2024-09-13T11:55:02.006 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:02.006 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:02.006 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.006 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.006 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:02.006 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.006 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:02.006 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:02.006 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:02.007 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.981+0000 7f8d6d7fa640 1 -- 172.21.15.100:0/2687184121 <== mon.2 v2:172.21.15.100:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f8d64013740 con 0x7f8d70138020 2024-09-13T11:55:02.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.981+0000 7f8d6d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:55:02.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.981+0000 7f8d6d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:55:02.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.981+0000 7f8d6d7fa640 1 -- 172.21.15.100:0/2687184121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d70138020 msgr2=0x7f8d7013aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.982+0000 7f8d6d7fa640 1 --2- 172.21.15.100:0/2687184121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d70138020 0x7f8d7013aed0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8d64000f60 tx=0x7f8d64000ed0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.982+0000 7f8d6d7fa640 1 -- 172.21.15.100:0/2687184121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d70138020 msgr2=0x7f8d7013aed0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:55:02.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.982+0000 7f8d6d7fa640 1 --2- 172.21.15.100:0/2687184121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d70138020 0x7f8d7013aed0 secure :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8d64000f60 tx=0x7f8d64000ed0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.008 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:55:02.008 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:55:02.008 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:55:02.008 INFO:teuthology.orchestra.run.smithi100.stdout: "state": "peon", 2024-09-13T11:55:02.008 INFO:teuthology.orchestra.run.smithi100.stdout: "election_epoch": 16, 2024-09-13T11:55:02.008 INFO:teuthology.orchestra.run.smithi100.stdout: "uptime": 141051, 2024-09-13T11:55:02.008 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum": [ 2024-09-13T11:55:02.008 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-13T11:55:02.008 INFO:teuthology.orchestra.run.smithi100.stdout: 1, 2024-09-13T11:55:02.008 INFO:teuthology.orchestra.run.smithi100.stdout: 2 2024-09-13T11:55:02.009 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:02.009 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_age": 2, 2024-09-13T11:55:02.009 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:55:02.009 INFO:teuthology.orchestra.run.smithi100.stdout: "required_con": "2450521708007866372", 2024-09-13T11:55:02.009 INFO:teuthology.orchestra.run.smithi100.stdout: "required_mon": [ 2024-09-13T11:55:02.009 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:55:02.009 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:55:02.009 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:55:02.009 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:55:02.010 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:55:02.010 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:55:02.010 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:55:02.010 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:55:02.010 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:55:02.010 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:55:02.010 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:55:02.010 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:02.010 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:55:02.010 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_mon": [ 2024-09-13T11:55:02.010 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:55:02.010 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:55:02.010 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:55:02.011 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:55:02.011 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:55:02.011 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:55:02.011 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:55:02.011 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:55:02.011 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:55:02.011 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:55:02.011 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:55:02.011 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:02.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:02.011 INFO:teuthology.orchestra.run.smithi100.stdout: "outside_quorum": [], 2024-09-13T11:55:02.011 INFO:teuthology.orchestra.run.smithi100.stdout: "extra_probe_peers": [], 2024-09-13T11:55:02.012 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_provider": [], 2024-09-13T11:55:02.012 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": { 2024-09-13T11:55:02.012 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-09-13T11:55:02.012 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "0b1f4b4d-fad4-4dd3-86ac-a286d80846c0", 2024-09-13T11:55:02.012 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:55:02.012 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:55:02.012 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release": 19, 2024-09-13T11:55:02.012 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release_name": "squid", 2024-09-13T11:55:02.012 INFO:teuthology.orchestra.run.smithi100.stdout: "election_strategy": 1, 2024-09-13T11:55:02.012 INFO:teuthology.orchestra.run.smithi100.stdout: "disallowed_leaders: ": "", 2024-09-13T11:55:02.012 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false, 2024-09-13T11:55:02.012 INFO:teuthology.orchestra.run.smithi100.stdout: "tiebreaker_mon": "", 2024-09-13T11:55:02.013 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_ranks: ": "", 2024-09-13T11:55:02.013 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:55:02.013 INFO:teuthology.orchestra.run.smithi100.stdout: "persistent": [ 2024-09-13T11:55:02.013 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:55:02.013 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:55:02.013 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:55:02.013 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:55:02.013 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:55:02.013 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:55:02.013 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:55:02.013 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:55:02.013 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:55:02.014 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:55:02.014 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:55:02.014 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:02.014 INFO:teuthology.orchestra.run.smithi100.stdout: "optional": [] 2024-09-13T11:55:02.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:02.014 INFO:teuthology.orchestra.run.smithi100.stdout: "mons": [ 2024-09-13T11:55:02.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.014 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:55:02.014 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:55:02.014 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:55:02.014 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:55:02.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.015 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:55:02.015 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3300", 2024-09-13T11:55:02.015 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:02.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:02.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.015 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:55:02.015 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789", 2024-09-13T11:55:02.015 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:02.015 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:02.015 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:02.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:02.016 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789/0", 2024-09-13T11:55:02.016 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-13T11:55:02.016 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:55:02.016 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:55:02.016 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:55:02.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:02.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.016 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:55:02.016 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:55:02.016 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:55:02.016 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:55:02.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.017 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:55:02.017 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:3300", 2024-09-13T11:55:02.017 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:02.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:02.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.017 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:55:02.017 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789", 2024-09-13T11:55:02.017 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:02.017 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:02.018 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:02.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:02.018 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789/0", 2024-09-13T11:55:02.018 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.122:6789/0", 2024-09-13T11:55:02.018 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:55:02.018 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:55:02.018 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:55:02.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:02.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.018 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:55:02.018 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:55:02.019 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:55:02.019 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:55:02.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.019 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:55:02.019 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3301", 2024-09-13T11:55:02.019 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:02.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:02.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.019 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:55:02.019 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790", 2024-09-13T11:55:02.020 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:02.020 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:02.020 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:02.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:02.020 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790/0", 2024-09-13T11:55:02.020 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-13T11:55:02.020 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:55:02.020 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:55:02.020 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:55:02.020 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:02.020 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:02.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:02.021 INFO:teuthology.orchestra.run.smithi100.stdout: "feature_map": { 2024-09-13T11:55:02.021 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": [ 2024-09-13T11:55:02.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.021 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:55:02.021 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:55:02.021 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:55:02.021 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:02.021 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:02.021 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": [ 2024-09-13T11:55:02.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.022 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:55:02.022 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:55:02.022 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 3 2024-09-13T11:55:02.022 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:02.022 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:02.022 INFO:teuthology.orchestra.run.smithi100.stdout: "client": [ 2024-09-13T11:55:02.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.022 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:55:02.022 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:55:02.022 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:55:02.023 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:02.023 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:02.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:02.023 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false 2024-09-13T11:55:02.023 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:55:02.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.982+0000 7f8d77223640 1 -- 172.21.15.100:0/2687184121 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f8d5403dfa0 msgr2=0x7f8d54040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.982+0000 7f8d77223640 1 --2- 172.21.15.100:0/2687184121 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f8d5403dfa0 0x7f8d54040460 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c034000 tx=0x7f8d5c02e000 comp rx=0 tx=0).stop 2024-09-13T11:55:02.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.982+0000 7f8d77223640 10 monclient: shutdown 2024-09-13T11:55:02.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.982+0000 7f8d77223640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:55:02.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.982+0000 7f8d77223640 1 -- 172.21.15.100:0/2687184121 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d7013aaf0 msgr2=0x7f8d701375a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.982+0000 7f8d757c0640 1 -- 172.21.15.100:0/2687184121 reap_dead start 2024-09-13T11:55:02.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.982+0000 7f8d77223640 1 --2- 172.21.15.100:0/2687184121 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d7013aaf0 0x7f8d701375a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8d70070ef0 tx=0x7f8d6800cb90 comp rx=0 tx=0).stop 2024-09-13T11:55:02.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.982+0000 7f8d77223640 1 -- 172.21.15.100:0/2687184121 shutdown_connections 2024-09-13T11:55:02.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.982+0000 7f8d77223640 1 --2- 172.21.15.100:0/2687184121 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d7013aaf0 0x7f8d701375a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.982+0000 7f8d77223640 1 -- 172.21.15.100:0/2687184121 >> 172.21.15.100:0/2687184121 conn(0x7f8d700a83f0 msgr2=0x7f8d7009da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:55:02.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.982+0000 7f8d77223640 1 -- 172.21.15.100:0/2687184121 shutdown_connections 2024-09-13T11:55:02.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:01.982+0000 7f8d77223640 1 -- 172.21.15.100:0/2687184121 wait complete. 2024-09-13T11:55:02.026 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T11:55:02.026 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-13T11:55:02.027 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-13T11:55:02.124 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.123+0000 7f5bcadcd640 10 monclient: get_monmap_and_config 2024-09-13T11:55:02.124 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.123+0000 7f5bcadcd640 10 monclient: build_initial_monmap 2024-09-13T11:55:02.124 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.123+0000 7f5bcadcd640 10 monclient: monmap: 2024-09-13T11:55:02.125 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:55:02.125 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:02.125 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:55:02.124156+0000 2024-09-13T11:55:02.125 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:55:02.124156+0000 2024-09-13T11:55:02.125 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:55:02.125 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.125 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:55:02.125 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:55:02.126 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:55:02.126 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:55:02.126 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:55:02.126 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.123+0000 7f5bcadcd640 1 Processor -- start 2024-09-13T11:55:02.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bcadcd640 1 -- start start 2024-09-13T11:55:02.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bcadcd640 10 monclient: init 2024-09-13T11:55:02.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bcadcd640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:02.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bcadcd640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-13T11:55:02.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bcadcd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5bc40a15a0 0x7f5bc40a1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bcadcd640 10 monclient(hunting): picked mon.noname-c con 0x7f5bc40a15a0 addr v2:172.21.15.122:3300/0 2024-09-13T11:55:02.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bcadcd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5bc409f6d0 addr v1:172.21.15.100:6789/0 2024-09-13T11:55:02.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bcadcd640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5bc40a1ec0 0x7f5bc4134ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bcadcd640 10 monclient(hunting): picked mon.noname-a con 0x7f5bc40a1ec0 addr v2:172.21.15.100:3300/0 2024-09-13T11:55:02.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bcadcd640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bc409ecf0 con 0x7f5bc409f6d0 2024-09-13T11:55:02.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bcadcd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:02.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bcadcd640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f5bc409d3b0 con 0x7f5bc40a1ec0 2024-09-13T11:55:02.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bcadcd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:02.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bcadcd640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f5bc409ea10 con 0x7f5bc40a15a0 2024-09-13T11:55:02.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bcadcd640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:02.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bcadcd640 10 monclient(hunting): authenticate will time out at 882.717617s 2024-09-13T11:55:02.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bc936a640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5bc40a1ec0 0x7f5bc4134ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:02.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bc3fff640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f5bc409f6d0 0x7f5bc41343d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:48288/0 (socket says 172.21.15.100:48288) 2024-09-13T11:55:02.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bc3fff640 1 -- 172.21.15.100:0/1485662392 learned_addr learned my addr 172.21.15.100:0/1485662392 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:55:02.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bc936a640 10 monclient(hunting): get_auth_request con 0x7f5bc40a1ec0 auth_method 0 2024-09-13T11:55:02.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bc936a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:02.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bc936a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:02.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bc936a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:02.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bc936a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:02.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bc936a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:02.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bc936a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:02.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bc8b69640 1 --2- 172.21.15.100:0/1485662392 >> v2:172.21.15.122:3300/0 conn(0x7f5bc40a15a0 0x7f5bc40a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:02.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bc8b69640 10 monclient(hunting): get_auth_request con 0x7f5bc40a15a0 auth_method 0 2024-09-13T11:55:02.138 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bc8b69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:02.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bc8b69640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:02.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bc8b69640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:02.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bc936a640 10 monclient(hunting): handle_auth_done global_id 14181 payload 274 2024-09-13T11:55:02.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bc936a640 1 -- 172.21.15.100:0/1485662392 >> v2:172.21.15.122:3300/0 conn(0x7f5bc40a15a0 msgr2=0x7f5bc40a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:55:02.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.124+0000 7f5bc936a640 1 --2- 172.21.15.100:0/1485662392 >> v2:172.21.15.122:3300/0 conn(0x7f5bc40a15a0 0x7f5bc40a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bc936a640 1 -- 172.21.15.100:0/1485662392 >> v1:172.21.15.100:6789/0 conn(0x7f5bc409f6d0 legacy=0x7f5bc41343d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bc936a640 10 monclient: _finish_hunting 0 2024-09-13T11:55:02.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bc936a640 1 monclient: found mon.noname-a 2024-09-13T11:55:02.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bc37fe640 1 -- 172.21.15.100:0/1485662392 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1013059405 0 0) 0x7f5bc409ecf0 con 0x7f5bc409f6d0 2024-09-13T11:55:02.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bc936a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:02.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bc936a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:55:02.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bc936a640 1 -- 172.21.15.100:0/1485662392 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc40a2330 con 0x7f5bc40a1ec0 2024-09-13T11:55:02.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bc37fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:55:02.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bc936a640 1 --2- 172.21.15.100:0/1485662392 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5bc40a1ec0 0x7f5bc4134ae0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5bc40a3580 tx=0x7f5bbc02a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d9b56828c2c6ae4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:02.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bc37fe640 1 -- 172.21.15.100:0/1485662392 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc007740 con 0x7f5bc40a1ec0 2024-09-13T11:55:02.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bc37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:02.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bc37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:55:02.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bc37fe640 10 monclient: dump: 2024-09-13T11:55:02.140 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:02.140 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:02.141 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.141 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.141 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:02.141 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.141 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:02.141 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:02.141 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:02.141 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bc37fe640 10 monclient: _finish_auth 0 2024-09-13T11:55:02.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bc37fe640 10 monclient: _check_auth_tickets 2024-09-13T11:55:02.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:02.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bc37fe640 1 -- 172.21.15.100:0/1485662392 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bbc0078e0 con 0x7f5bc40a1ec0 2024-09-13T11:55:02.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bcadcd640 5 monclient: authenticate success, global_id 14181 2024-09-13T11:55:02.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bcadcd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:55:02.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bc37fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:55:02.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bcadcd640 10 monclient: get_monmap_and_config success 2024-09-13T11:55:02.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bc37fe640 1 -- 172.21.15.100:0/1485662392 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc007cb0 con 0x7f5bc40a1ec0 2024-09-13T11:55:02.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bc37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:02.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bcadcd640 4 set_mon_vals no callback set 2024-09-13T11:55:02.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bc37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:02.142 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.125+0000 7f5bc37fe640 10 monclient: dump: 2024-09-13T11:55:02.142 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:02.142 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:02.142 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.142 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.142 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:02.142 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.142 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:02.143 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:02.143 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:02.143 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.126+0000 7f5bcadcd640 10 monclient: shutdown 2024-09-13T11:55:02.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.126+0000 7f5bcadcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:55:02.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.126+0000 7f5bcadcd640 1 -- 172.21.15.100:0/1485662392 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5bc40a1ec0 msgr2=0x7f5bc4134ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.126+0000 7f5bcadcd640 1 --2- 172.21.15.100:0/1485662392 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5bc40a1ec0 0x7f5bc4134ae0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5bc40a3580 tx=0x7f5bbc02a560 comp rx=0 tx=0).stop 2024-09-13T11:55:02.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.126+0000 7f5bcadcd640 1 -- 172.21.15.100:0/1485662392 shutdown_connections 2024-09-13T11:55:02.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.126+0000 7f5bcadcd640 1 --2- 172.21.15.100:0/1485662392 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5bc40a1ec0 0x7f5bc4134ae0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.126+0000 7f5bcadcd640 1 --2- 172.21.15.100:0/1485662392 >> v2:172.21.15.122:3300/0 conn(0x7f5bc40a15a0 0x7f5bc40a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.126+0000 7f5bcadcd640 1 -- 172.21.15.100:0/1485662392 >> 172.21.15.100:0/1485662392 conn(0x7f5bc40a6470 msgr2=0x7f5bc40a2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:55:02.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.126+0000 7f5bcadcd640 1 -- 172.21.15.100:0/1485662392 shutdown_connections 2024-09-13T11:55:02.143 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.126+0000 7f5bcadcd640 1 -- 172.21.15.100:0/1485662392 wait complete. 2024-09-13T11:55:02.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.126+0000 7f5bcadcd640 10 monclient: build_initial_monmap 2024-09-13T11:55:02.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.126+0000 7f5bcadcd640 10 monclient: monmap: 2024-09-13T11:55:02.144 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:55:02.144 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:55:02.144 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:55:02.144 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:55:02.144 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:55:02.144 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.144 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:55:02.144 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:55:02.144 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:55:02.144 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.127+0000 7f5bcadcd640 1 Processor -- start 2024-09-13T11:55:02.144 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.127+0000 7f5bcadcd640 1 -- start start 2024-09-13T11:55:02.145 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.127+0000 7f5bcadcd640 10 monclient: init 2024-09-13T11:55:02.145 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.127+0000 7f5bcadcd640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:02.145 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.127+0000 7f5bcadcd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:55:02.145 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.127+0000 7f5bcadcd640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5bc409f6d0 0x7f5bc411db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.145 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.127+0000 7f5bcadcd640 10 monclient(hunting): picked mon.noname-a con 0x7f5bc409f6d0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:55:02.145 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.127+0000 7f5bcadcd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc40a15a0 0x7f5bc411e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.145 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.127+0000 7f5bcadcd640 10 monclient(hunting): picked mon.noname-b con 0x7f5bc40a15a0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:55:02.145 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.127+0000 7f5bcadcd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5bc40a1ec0 0x7f5bc411e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.145 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.127+0000 7f5bcadcd640 10 monclient(hunting): picked mon.noname-c con 0x7f5bc40a1ec0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:55:02.145 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.127+0000 7f5bcadcd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:02.145 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.127+0000 7f5bcadcd640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc41351a0 con 0x7f5bc409f6d0 2024-09-13T11:55:02.145 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.127+0000 7f5bcadcd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:02.145 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.127+0000 7f5bcadcd640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc4135020 con 0x7f5bc40a15a0 2024-09-13T11:55:02.146 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.127+0000 7f5bcadcd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:02.146 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.127+0000 7f5bcadcd640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f5bc409cfa0 con 0x7f5bc40a1ec0 2024-09-13T11:55:02.146 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.127+0000 7f5bcadcd640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:02.146 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.127+0000 7f5bcadcd640 10 monclient(hunting): authenticate will time out at 882.720964s 2024-09-13T11:55:02.146 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc8b69640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5bc409f6d0 0x7f5bc411db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:02.146 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc8b69640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5bc409f6d0 0x7f5bc411db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53502/0 (socket says 172.21.15.100:53502) 2024-09-13T11:55:02.146 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc8b69640 1 -- 172.21.15.100:0/99050374 learned_addr learned my addr 172.21.15.100:0/99050374 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:55:02.146 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc8b69640 10 monclient(hunting): get_auth_request con 0x7f5bc409f6d0 auth_method 0 2024-09-13T11:55:02.146 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc8b69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:02.146 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc8b69640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:02.146 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc8b69640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:02.146 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc936a640 1 --2- 172.21.15.100:0/99050374 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5bc40a1ec0 0x7f5bc411e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:02.146 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc936a640 10 monclient(hunting): get_auth_request con 0x7f5bc40a1ec0 auth_method 0 2024-09-13T11:55:02.146 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc936a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:02.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc936a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:02.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc936a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:02.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc8b69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:02.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc8b69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:02.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc8b69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:02.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc936a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:02.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc936a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:02.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc936a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:02.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc8b69640 10 monclient(hunting): handle_auth_done global_id 14187 payload 824 2024-09-13T11:55:02.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc3fff640 1 --2- 172.21.15.100:0/99050374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc40a15a0 0x7f5bc411e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:02.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc8b69640 1 -- 172.21.15.100:0/99050374 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5bc40a1ec0 msgr2=0x7f5bc411e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc8b69640 1 --2- 172.21.15.100:0/99050374 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5bc40a1ec0 0x7f5bc411e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc8b69640 1 -- 172.21.15.100:0/99050374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc40a15a0 msgr2=0x7f5bc411e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc8b69640 1 --2- 172.21.15.100:0/99050374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc40a15a0 0x7f5bc411e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc8b69640 10 monclient: _finish_hunting 0 2024-09-13T11:55:02.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc8b69640 1 monclient: found mon.noname-a 2024-09-13T11:55:02.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc8b69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:02.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc8b69640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:55:02.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc8b69640 1 -- 172.21.15.100:0/99050374 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc41215f0 con 0x7f5bc409f6d0 2024-09-13T11:55:02.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc8b69640 1 --2- 172.21.15.100:0/99050374 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5bc409f6d0 0x7f5bc411db00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5bb800d830 tx=0x7f5bb800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:02.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc17fa640 1 -- 172.21.15.100:0/99050374 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb800b6c0 con 0x7f5bc409f6d0 2024-09-13T11:55:02.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:02.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:55:02.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc17fa640 10 monclient: dump: 2024-09-13T11:55:02.148 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:02.148 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:02.148 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.148 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.149 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:02.149 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.149 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:02.149 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:02.149 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:02.149 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc17fa640 10 monclient: _finish_auth 0 2024-09-13T11:55:02.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc17fa640 10 monclient: _check_auth_tickets 2024-09-13T11:55:02.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:02.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bcadcd640 5 monclient: authenticate success, global_id 14187 2024-09-13T11:55:02.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc17fa640 1 -- 172.21.15.100:0/99050374 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bb8019070 con 0x7f5bc409f6d0 2024-09-13T11:55:02.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc17fa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:55:02.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc17fa640 1 -- 172.21.15.100:0/99050374 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb800bac0 con 0x7f5bc409f6d0 2024-09-13T11:55:02.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bcadcd640 10 monclient: _renew_subs 2024-09-13T11:55:02.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bcadcd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:02.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bcadcd640 1 -- 172.21.15.100:0/99050374 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bc41218d0 con 0x7f5bc409f6d0 2024-09-13T11:55:02.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc2ffd640 4 set_mon_vals no callback set 2024-09-13T11:55:02.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:02.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:02.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bc17fa640 10 monclient: dump: 2024-09-13T11:55:02.150 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:02.150 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:02.150 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.150 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.150 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:02.150 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.150 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:02.150 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:02.151 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:02.151 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bcadcd640 10 monclient: _renew_subs 2024-09-13T11:55:02.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bcadcd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:02.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.128+0000 7f5bcadcd640 1 -- 172.21.15.100:0/99050374 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bc4120a60 con 0x7f5bc409f6d0 2024-09-13T11:55:02.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.129+0000 7f5bc17fa640 1 -- 172.21.15.100:0/99050374 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f5bb801e460 con 0x7f5bc409f6d0 2024-09-13T11:55:02.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.129+0000 7f5bcadcd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:02.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.129+0000 7f5bcadcd640 1 --2- 172.21.15.100:0/99050374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc41c8450 0x7f5bc41c8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.129+0000 7f5bcadcd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:02.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.129+0000 7f5bcadcd640 1 -- 172.21.15.100:0/99050374 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc4134400 con 0x7f5bc41c8450 2024-09-13T11:55:02.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.129+0000 7f5bc17fa640 1 --2- 172.21.15.100:0/99050374 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f5ba803dfa0 0x7f5ba8040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.129+0000 7f5bc17fa640 1 -- 172.21.15.100:0/99050374 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4346+0+0 (secure 0 0 0) 0x7f5bb8052c10 con 0x7f5bc409f6d0 2024-09-13T11:55:02.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.129+0000 7f5bc3fff640 1 --2- 172.21.15.100:0/99050374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc41c8450 0x7f5bc41c8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:02.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.129+0000 7f5bc3fff640 10 monclient: get_auth_request con 0x7f5bc41c8450 auth_method 0 2024-09-13T11:55:02.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.129+0000 7f5bc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:02.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.129+0000 7f5bc3fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:02.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.129+0000 7f5bc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:02.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.129+0000 7f5bc936a640 1 --2- 172.21.15.100:0/99050374 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f5ba803dfa0 0x7f5ba8040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:02.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.129+0000 7f5bc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:02.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.129+0000 7f5bc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:02.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.129+0000 7f5bc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:02.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.129+0000 7f5bc936a640 10 monclient: get_auth_request con 0x7f5ba803dfa0 auth_method 0 2024-09-13T11:55:02.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.130+0000 7f5bc3fff640 10 monclient(hunting): handle_auth_done global_id 4987 payload 824 2024-09-13T11:55:02.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.130+0000 7f5bc3fff640 1 -- 172.21.15.100:0/99050374 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5bc40a1980 con 0x7f5bc41c8450 2024-09-13T11:55:02.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.130+0000 7f5bc3fff640 1 --2- 172.21.15.100:0/99050374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc41c8450 0x7f5bc41c8830 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0000f60 tx=0x7f5bb000d8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:02.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.130+0000 7f5bc936a640 1 --2- 172.21.15.100:0/99050374 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f5ba803dfa0 0x7f5ba8040460 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc034000 tx=0x7f5bbc02f000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:02.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.130+0000 7f5bc17fa640 1 -- 172.21.15.100:0/99050374 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb00074e0 con 0x7f5bc41c8450 2024-09-13T11:55:02.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.130+0000 7f5bc17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:02.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.130+0000 7f5bc17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:55:02.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.130+0000 7f5bc17fa640 10 monclient: dump: 2024-09-13T11:55:02.153 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:02.153 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:02.153 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.153 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.153 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:02.153 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.154 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:02.154 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:02.154 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:02.154 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.130+0000 7f5bc17fa640 1 -- 172.21.15.100:0/99050374 <== mon.1 v2:172.21.15.122:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5bb00074e0 con 0x7f5bc41c8450 2024-09-13T11:55:02.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.130+0000 7f5bc17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:02.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.130+0000 7f5bc17fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:55:02.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.130+0000 7f5bc17fa640 1 -- 172.21.15.100:0/99050374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc41c8450 msgr2=0x7f5bc41c8830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.130+0000 7f5bc17fa640 1 --2- 172.21.15.100:0/99050374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc41c8450 0x7f5bc41c8830 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0000f60 tx=0x7f5bb000d8a0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.130+0000 7f5bc17fa640 1 -- 172.21.15.100:0/99050374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc41c8450 msgr2=0x7f5bc41c8830 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:55:02.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.130+0000 7f5bc17fa640 1 --2- 172.21.15.100:0/99050374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc41c8450 0x7f5bc41c8830 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.136+0000 7f5bcadcd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:55:02.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.136+0000 7f5bcadcd640 1 --2- 172.21.15.100:0/99050374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc41c8d70 0x7f5bc4071200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.136+0000 7f5bcadcd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:02.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.136+0000 7f5bcadcd640 1 -- 172.21.15.100:0/99050374 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc40008d0 con 0x7f5bc41c8d70 2024-09-13T11:55:02.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.137+0000 7f5bc3fff640 1 --2- 172.21.15.100:0/99050374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc41c8d70 0x7f5bc4071200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:02.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.137+0000 7f5bc3fff640 10 monclient: get_auth_request con 0x7f5bc41c8d70 auth_method 0 2024-09-13T11:55:02.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.137+0000 7f5bc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:02.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.137+0000 7f5bc3fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:02.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.137+0000 7f5bc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:02.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.137+0000 7f5bc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:02.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.137+0000 7f5bc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:02.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.137+0000 7f5bc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:02.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.137+0000 7f5bc3fff640 10 monclient(hunting): handle_auth_done global_id 4990 payload 824 2024-09-13T11:55:02.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.137+0000 7f5bc3fff640 1 -- 172.21.15.100:0/99050374 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5bc4121cf0 con 0x7f5bc41c8d70 2024-09-13T11:55:02.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.137+0000 7f5bc3fff640 1 --2- 172.21.15.100:0/99050374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc41c8d70 0x7f5bc4071200 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0000f60 tx=0x7f5bb0001080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:02.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.137+0000 7f5bc17fa640 1 -- 172.21.15.100:0/99050374 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb001bc30 con 0x7f5bc41c8d70 2024-09-13T11:55:02.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.137+0000 7f5bc17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:02.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.137+0000 7f5bc17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:55:02.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.137+0000 7f5bc17fa640 10 monclient: dump: 2024-09-13T11:55:02.157 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:02.157 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:02.158 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.158 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.158 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:02.158 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.158 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:02.158 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:02.158 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:02.158 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.137+0000 7f5bc17fa640 1 -- 172.21.15.100:0/99050374 <== mon.1 v2:172.21.15.122:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f5bb0013420 con 0x7f5bc41c8d70 2024-09-13T11:55:02.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.137+0000 7f5bc17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:55:02.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.137+0000 7f5bc17fa640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:55:02.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.137+0000 7f5bc17fa640 1 -- 172.21.15.100:0/99050374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc41c8d70 msgr2=0x7f5bc4071200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.137+0000 7f5bc17fa640 1 --2- 172.21.15.100:0/99050374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc41c8d70 0x7f5bc4071200 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0000f60 tx=0x7f5bb0001080 comp rx=0 tx=0).stop 2024-09-13T11:55:02.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.137+0000 7f5bc17fa640 1 -- 172.21.15.100:0/99050374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc41c8d70 msgr2=0x7f5bc4071200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:55:02.159 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.137+0000 7f5bc17fa640 1 --2- 172.21.15.100:0/99050374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc41c8d70 0x7f5bc4071200 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.159 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:55:02.159 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:55:02.159 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:55:02.159 INFO:teuthology.orchestra.run.smithi100.stdout: "state": "peon", 2024-09-13T11:55:02.159 INFO:teuthology.orchestra.run.smithi100.stdout: "election_epoch": 16, 2024-09-13T11:55:02.159 INFO:teuthology.orchestra.run.smithi100.stdout: "uptime": 141167, 2024-09-13T11:55:02.159 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum": [ 2024-09-13T11:55:02.159 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-13T11:55:02.159 INFO:teuthology.orchestra.run.smithi100.stdout: 1, 2024-09-13T11:55:02.160 INFO:teuthology.orchestra.run.smithi100.stdout: 2 2024-09-13T11:55:02.160 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:02.160 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_age": 2, 2024-09-13T11:55:02.160 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:55:02.160 INFO:teuthology.orchestra.run.smithi100.stdout: "required_con": "2450521708007866372", 2024-09-13T11:55:02.160 INFO:teuthology.orchestra.run.smithi100.stdout: "required_mon": [ 2024-09-13T11:55:02.160 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:55:02.160 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:55:02.160 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:55:02.160 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:55:02.160 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:55:02.160 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:55:02.160 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:55:02.160 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:55:02.161 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:55:02.161 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:55:02.161 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:55:02.161 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:02.161 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:55:02.161 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_mon": [ 2024-09-13T11:55:02.161 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:55:02.161 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:55:02.161 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:55:02.161 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:55:02.161 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:55:02.161 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:55:02.161 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:55:02.161 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:55:02.162 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:55:02.162 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:55:02.162 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:55:02.162 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:02.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:02.162 INFO:teuthology.orchestra.run.smithi100.stdout: "outside_quorum": [], 2024-09-13T11:55:02.162 INFO:teuthology.orchestra.run.smithi100.stdout: "extra_probe_peers": [], 2024-09-13T11:55:02.162 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_provider": [], 2024-09-13T11:55:02.162 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": { 2024-09-13T11:55:02.162 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-09-13T11:55:02.162 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "0b1f4b4d-fad4-4dd3-86ac-a286d80846c0", 2024-09-13T11:55:02.162 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:55:02.162 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:55:02.162 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release": 19, 2024-09-13T11:55:02.163 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release_name": "squid", 2024-09-13T11:55:02.163 INFO:teuthology.orchestra.run.smithi100.stdout: "election_strategy": 1, 2024-09-13T11:55:02.163 INFO:teuthology.orchestra.run.smithi100.stdout: "disallowed_leaders: ": "", 2024-09-13T11:55:02.163 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false, 2024-09-13T11:55:02.163 INFO:teuthology.orchestra.run.smithi100.stdout: "tiebreaker_mon": "", 2024-09-13T11:55:02.163 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_ranks: ": "", 2024-09-13T11:55:02.163 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:55:02.163 INFO:teuthology.orchestra.run.smithi100.stdout: "persistent": [ 2024-09-13T11:55:02.163 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:55:02.163 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:55:02.163 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:55:02.163 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:55:02.163 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:55:02.163 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:55:02.164 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:55:02.164 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:55:02.164 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:55:02.164 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:55:02.164 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:55:02.164 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:02.164 INFO:teuthology.orchestra.run.smithi100.stdout: "optional": [] 2024-09-13T11:55:02.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:02.164 INFO:teuthology.orchestra.run.smithi100.stdout: "mons": [ 2024-09-13T11:55:02.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.164 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:55:02.164 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:55:02.164 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:55:02.165 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:55:02.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.165 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:55:02.165 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3300", 2024-09-13T11:55:02.165 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:02.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:02.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.165 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:55:02.165 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789", 2024-09-13T11:55:02.165 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:02.165 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:02.165 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:02.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:02.165 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789/0", 2024-09-13T11:55:02.166 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-13T11:55:02.166 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:55:02.166 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:55:02.166 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:55:02.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:02.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.166 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:55:02.166 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:55:02.166 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:55:02.166 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:55:02.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.166 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:55:02.166 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:3300", 2024-09-13T11:55:02.166 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:02.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:02.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.167 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:55:02.167 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789", 2024-09-13T11:55:02.167 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:02.167 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:02.167 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:02.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:02.167 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789/0", 2024-09-13T11:55:02.167 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.122:6789/0", 2024-09-13T11:55:02.167 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:55:02.167 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:55:02.167 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:55:02.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:02.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.168 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:55:02.168 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:55:02.168 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:55:02.168 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:55:02.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.168 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:55:02.168 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3301", 2024-09-13T11:55:02.168 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:02.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:02.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.168 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:55:02.168 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790", 2024-09-13T11:55:02.169 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:02.169 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:02.169 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:02.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:02.169 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790/0", 2024-09-13T11:55:02.169 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-13T11:55:02.169 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:55:02.169 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:55:02.169 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:55:02.169 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:02.169 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:02.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:02.169 INFO:teuthology.orchestra.run.smithi100.stdout: "feature_map": { 2024-09-13T11:55:02.169 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": [ 2024-09-13T11:55:02.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.170 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:55:02.170 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:55:02.170 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:55:02.170 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:02.170 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:02.170 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": [ 2024-09-13T11:55:02.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.170 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:55:02.170 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:55:02.170 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 3 2024-09-13T11:55:02.170 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:02.170 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:02.170 INFO:teuthology.orchestra.run.smithi100.stdout: "client": [ 2024-09-13T11:55:02.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.171 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:55:02.171 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:55:02.171 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 6 2024-09-13T11:55:02.171 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:02.171 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:02.171 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": [ 2024-09-13T11:55:02.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:02.171 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:55:02.171 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:55:02.171 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:55:02.171 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:02.171 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:02.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:02.171 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false 2024-09-13T11:55:02.172 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:55:02.173 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.138+0000 7f5bcadcd640 1 -- 172.21.15.100:0/99050374 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f5ba803dfa0 msgr2=0x7f5ba8040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.173 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.138+0000 7f5bcadcd640 1 --2- 172.21.15.100:0/99050374 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f5ba803dfa0 0x7f5ba8040460 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc034000 tx=0x7f5bbc02f000 comp rx=0 tx=0).stop 2024-09-13T11:55:02.173 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.138+0000 7f5bcadcd640 10 monclient: shutdown 2024-09-13T11:55:02.173 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.138+0000 7f5bcadcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:55:02.173 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.138+0000 7f5bcadcd640 1 -- 172.21.15.100:0/99050374 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5bc409f6d0 msgr2=0x7f5bc411db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.173 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.138+0000 7f5bcadcd640 1 --2- 172.21.15.100:0/99050374 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5bc409f6d0 0x7f5bc411db00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5bb800d830 tx=0x7f5bb800dcf0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.138+0000 7f5bc936a640 1 -- 172.21.15.100:0/99050374 reap_dead start 2024-09-13T11:55:02.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.138+0000 7f5bc936a640 1 -- 172.21.15.100:0/99050374 reap_dead start 2024-09-13T11:55:02.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.138+0000 7f5bcadcd640 1 -- 172.21.15.100:0/99050374 shutdown_connections 2024-09-13T11:55:02.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.138+0000 7f5bcadcd640 1 -- 172.21.15.100:0/99050374 >> 172.21.15.100:0/99050374 conn(0x7f5bc40a6470 msgr2=0x7f5bc40a4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:55:02.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.138+0000 7f5bcadcd640 1 -- 172.21.15.100:0/99050374 shutdown_connections 2024-09-13T11:55:02.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.138+0000 7f5bcadcd640 1 -- 172.21.15.100:0/99050374 wait complete. 2024-09-13T11:55:02.174 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T11:55:02.174 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-13T11:55:02.174 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-13T11:55:02.175 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-13T11:55:02.175 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c store 2024-09-13T11:55:02.175 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c sync_force --yes-i-really-mean-it 2024-09-13T11:55:02.273 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.272+0000 7ff8ff75d640 10 monclient: get_monmap_and_config 2024-09-13T11:55:02.273 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.272+0000 7ff8ff75d640 10 monclient: build_initial_monmap 2024-09-13T11:55:02.274 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.272+0000 7ff8ff75d640 10 monclient: monmap: 2024-09-13T11:55:02.274 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:55:02.274 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:02.274 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:55:02.273128+0000 2024-09-13T11:55:02.274 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:55:02.273128+0000 2024-09-13T11:55:02.274 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:55:02.274 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.274 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:55:02.274 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:55:02.275 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:55:02.275 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:55:02.275 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:55:02.275 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.285 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.272+0000 7ff8ff75d640 1 Processor -- start 2024-09-13T11:55:02.285 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8ff75d640 1 -- start start 2024-09-13T11:55:02.285 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8ff75d640 10 monclient: init 2024-09-13T11:55:02.285 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8ff75d640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:02.285 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8ff75d640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-13T11:55:02.285 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8ff75d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff8f8137770 addr v1:172.21.15.122:6789/0 2024-09-13T11:55:02.285 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8ff75d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff8f809ca70 addr v1:172.21.15.100:6789/0 2024-09-13T11:55:02.285 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8ff75d640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff8f8076ba0 0x7ff8f813fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.286 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8ff75d640 10 monclient(hunting): picked mon.noname-a con 0x7ff8f8076ba0 addr v2:172.21.15.100:3300/0 2024-09-13T11:55:02.286 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8ff75d640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8f809c630 con 0x7ff8f809ca70 2024-09-13T11:55:02.286 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8ff75d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8f80a4460 con 0x7ff8f8137770 2024-09-13T11:55:02.286 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8ff75d640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:02.286 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8ff75d640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7ff8f80a42a0 con 0x7ff8f8076ba0 2024-09-13T11:55:02.286 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8ff75d640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:02.286 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8ff75d640 10 monclient(hunting): authenticate will time out at 882.866594s 2024-09-13T11:55:02.286 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8fdcfa640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff8f8076ba0 0x7ff8f813fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:02.286 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8fccf8640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7ff8f809ca70 0x7ff8f813c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:35540/0 (socket says 172.21.15.100:35540) 2024-09-13T11:55:02.286 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8fccf8640 1 -- 172.21.15.100:0/3509192993 learned_addr learned my addr 172.21.15.100:0/3509192993 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:55:02.286 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8fdcfa640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff8f8076ba0 0x7ff8f813fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:46992/0 (socket says 172.21.15.100:46992) 2024-09-13T11:55:02.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8fdcfa640 10 monclient(hunting): get_auth_request con 0x7ff8f8076ba0 auth_method 0 2024-09-13T11:55:02.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8fdcfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:02.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8fdcfa640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:02.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8fdcfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:02.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8fdcfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:02.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8fdcfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:02.287 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8fdcfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:02.288 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8fdcfa640 10 monclient(hunting): handle_auth_done global_id 14190 payload 274 2024-09-13T11:55:02.288 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8fdcfa640 1 -- 172.21.15.100:0/3509192993 >> v1:172.21.15.122:6789/0 conn(0x7ff8f8137770 legacy=0x7ff8f8137b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.288 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8fdcfa640 1 -- 172.21.15.100:0/3509192993 >> v1:172.21.15.100:6789/0 conn(0x7ff8f809ca70 legacy=0x7ff8f813c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.288 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8fdcfa640 10 monclient: _finish_hunting 0 2024-09-13T11:55:02.288 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8fdcfa640 1 monclient: found mon.noname-a 2024-09-13T11:55:02.288 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8fdcfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:02.288 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.274+0000 7ff8fdcfa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:55:02.288 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.274+0000 7ff8fdcfa640 1 -- 172.21.15.100:0/3509192993 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8f8076300 con 0x7ff8f8076ba0 2024-09-13T11:55:02.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.273+0000 7ff8f7fff640 1 -- 172.21.15.100:0/3509192993 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3704497545 0 0) 0x7ff8f809c630 con 0x7ff8f809ca70 2024-09-13T11:55:02.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.274+0000 7ff8f7fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:55:02.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.274+0000 7ff8fdcfa640 1 --2- 172.21.15.100:0/3509192993 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8f8076ba0 0x7ff8f813fe00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff8e80097a0 tx=0x7ff8e802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c89dc70d3a7a8f07 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:02.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.274+0000 7ff8f7fff640 1 -- 172.21.15.100:0/3509192993 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8037070 con 0x7ff8f8076ba0 2024-09-13T11:55:02.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.274+0000 7ff8f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:02.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.274+0000 7ff8f7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:55:02.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.274+0000 7ff8f7fff640 10 monclient: dump: 2024-09-13T11:55:02.289 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:02.289 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:02.289 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.290 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.290 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:02.290 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.290 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:02.290 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:02.290 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:02.290 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.274+0000 7ff8f7fff640 10 monclient: _finish_auth 0 2024-09-13T11:55:02.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.274+0000 7ff8f7fff640 10 monclient: _check_auth_tickets 2024-09-13T11:55:02.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.274+0000 7ff8f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:02.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.274+0000 7ff8f7fff640 1 -- 172.21.15.100:0/3509192993 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8e8032440 con 0x7ff8f8076ba0 2024-09-13T11:55:02.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.274+0000 7ff8ff75d640 5 monclient: authenticate success, global_id 14190 2024-09-13T11:55:02.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.274+0000 7ff8ff75d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:55:02.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.274+0000 7ff8f7fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:55:02.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.274+0000 7ff8ff75d640 10 monclient: get_monmap_and_config success 2024-09-13T11:55:02.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.274+0000 7ff8f7fff640 1 -- 172.21.15.100:0/3509192993 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e802ae60 con 0x7ff8f8076ba0 2024-09-13T11:55:02.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.274+0000 7ff8f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:02.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.274+0000 7ff8ff75d640 4 set_mon_vals no callback set 2024-09-13T11:55:02.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.274+0000 7ff8f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:02.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.274+0000 7ff8f7fff640 10 monclient: dump: 2024-09-13T11:55:02.291 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:02.292 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:02.292 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.292 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.292 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:02.292 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.292 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:02.292 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:02.292 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:02.292 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.275+0000 7ff8ff75d640 10 monclient: shutdown 2024-09-13T11:55:02.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.275+0000 7ff8ff75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:55:02.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.275+0000 7ff8ff75d640 1 -- 172.21.15.100:0/3509192993 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8f8076ba0 msgr2=0x7ff8f813fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.275+0000 7ff8ff75d640 1 --2- 172.21.15.100:0/3509192993 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8f8076ba0 0x7ff8f813fe00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff8e80097a0 tx=0x7ff8e802a110 comp rx=0 tx=0).stop 2024-09-13T11:55:02.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.275+0000 7ff8ff75d640 1 -- 172.21.15.100:0/3509192993 shutdown_connections 2024-09-13T11:55:02.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.275+0000 7ff8ff75d640 1 --2- 172.21.15.100:0/3509192993 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8f8076ba0 0x7ff8f813fe00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.275+0000 7ff8ff75d640 1 -- 172.21.15.100:0/3509192993 >> 172.21.15.100:0/3509192993 conn(0x7ff8f80a7260 msgr2=0x7ff8f80a3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:55:02.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.275+0000 7ff8ff75d640 1 -- 172.21.15.100:0/3509192993 shutdown_connections 2024-09-13T11:55:02.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.275+0000 7ff8ff75d640 1 -- 172.21.15.100:0/3509192993 wait complete. 2024-09-13T11:55:02.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.275+0000 7ff8ff75d640 10 monclient: build_initial_monmap 2024-09-13T11:55:02.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.275+0000 7ff8ff75d640 10 monclient: monmap: 2024-09-13T11:55:02.294 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:55:02.294 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:55:02.294 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:55:02.294 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:55:02.294 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:55:02.294 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.294 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:55:02.294 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:55:02.294 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:55:02.294 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.295 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8ff75d640 1 Processor -- start 2024-09-13T11:55:02.295 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8ff75d640 1 -- start start 2024-09-13T11:55:02.295 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8ff75d640 10 monclient: init 2024-09-13T11:55:02.295 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8ff75d640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:02.295 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8ff75d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:55:02.295 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8ff75d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8f8076ba0 0x7ff8f811d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.295 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8ff75d640 10 monclient(hunting): picked mon.noname-b con 0x7ff8f8076ba0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:55:02.295 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8ff75d640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8f809ca70 0x7ff8f811d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.295 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8ff75d640 10 monclient(hunting): picked mon.noname-a con 0x7ff8f809ca70 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:55:02.295 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8ff75d640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8f8137770 0x7ff8f811dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.295 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8ff75d640 10 monclient(hunting): picked mon.noname-c con 0x7ff8f8137770 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:55:02.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8ff75d640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:02.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8ff75d640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f8076920 con 0x7ff8f809ca70 2024-09-13T11:55:02.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8ff75d640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:02.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8ff75d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f80767a0 con 0x7ff8f8076ba0 2024-09-13T11:55:02.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8ff75d640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:02.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8ff75d640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7ff8f8140340 con 0x7ff8f8137770 2024-09-13T11:55:02.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8ff75d640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:02.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8ff75d640 10 monclient(hunting): authenticate will time out at 882.869850s 2024-09-13T11:55:02.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8fccf8640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8f809ca70 0x7ff8f811d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:02.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8fccf8640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8f809ca70 0x7ff8f811d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:46994/0 (socket says 172.21.15.100:46994) 2024-09-13T11:55:02.296 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8fccf8640 1 -- 172.21.15.100:0/2213267148 learned_addr learned my addr 172.21.15.100:0/2213267148 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:55:02.297 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8fccf8640 10 monclient(hunting): get_auth_request con 0x7ff8f809ca70 auth_method 0 2024-09-13T11:55:02.297 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8fccf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:02.297 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8fccf8640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:02.297 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8fccf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:02.297 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8fdcfa640 1 --2- 172.21.15.100:0/2213267148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8f8137770 0x7ff8f811dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:02.297 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.276+0000 7ff8fdcfa640 10 monclient(hunting): get_auth_request con 0x7ff8f8137770 auth_method 0 2024-09-13T11:55:02.297 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fdcfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:02.297 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fdcfa640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:02.297 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fdcfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:02.297 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fccf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:02.297 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fccf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:02.297 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fccf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:02.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fd4f9640 1 --2- 172.21.15.100:0/2213267148 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8f8076ba0 0x7ff8f811d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:02.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fdcfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:02.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fdcfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:02.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fdcfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:02.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fd4f9640 10 monclient(hunting): get_auth_request con 0x7ff8f8076ba0 auth_method 0 2024-09-13T11:55:02.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fd4f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:02.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fd4f9640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:02.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fd4f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:02.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fd4f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:02.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fd4f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:02.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fd4f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:02.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fdcfa640 10 monclient(hunting): handle_auth_done global_id 4886 payload 824 2024-09-13T11:55:02.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fdcfa640 1 -- 172.21.15.100:0/2213267148 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8f8076ba0 msgr2=0x7ff8f811d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fdcfa640 1 --2- 172.21.15.100:0/2213267148 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8f8076ba0 0x7ff8f811d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fdcfa640 1 -- 172.21.15.100:0/2213267148 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8f809ca70 msgr2=0x7ff8f811d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fdcfa640 1 --2- 172.21.15.100:0/2213267148 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8f809ca70 0x7ff8f811d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fdcfa640 10 monclient: _finish_hunting 0 2024-09-13T11:55:02.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fdcfa640 1 monclient: found mon.noname-c 2024-09-13T11:55:02.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fdcfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:02.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fdcfa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:55:02.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fdcfa640 1 -- 172.21.15.100:0/2213267148 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8f8074f50 con 0x7ff8f8137770 2024-09-13T11:55:02.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fccf8640 1 --2- 172.21.15.100:0/2213267148 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8f809ca70 0x7ff8f811d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:55:02.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8fdcfa640 1 --2- 172.21.15.100:0/2213267148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8f8137770 0x7ff8f811dd00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8031000 tx=0x7ff8e80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:02.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8f5ffb640 1 -- 172.21.15.100:0/2213267148 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8037040 con 0x7ff8f8137770 2024-09-13T11:55:02.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:02.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8f5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:55:02.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8f5ffb640 10 monclient: dump: 2024-09-13T11:55:02.300 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:02.300 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:02.300 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.300 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.300 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:02.300 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.300 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:02.300 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:02.301 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:02.301 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8f5ffb640 10 monclient: _finish_auth 0 2024-09-13T11:55:02.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8f5ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:55:02.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:02.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8f5ffb640 1 -- 172.21.15.100:0/2213267148 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8e8042070 con 0x7ff8f8137770 2024-09-13T11:55:02.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8ff75d640 5 monclient: authenticate success, global_id 4886 2024-09-13T11:55:02.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8f5ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:55:02.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8f5ffb640 1 -- 172.21.15.100:0/2213267148 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8030c10 con 0x7ff8f8137770 2024-09-13T11:55:02.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:02.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8f5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:55:02.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8f5ffb640 10 monclient: dump: 2024-09-13T11:55:02.302 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:02.302 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:02.302 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.302 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.302 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:02.302 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.302 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:02.302 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:02.302 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:02.302 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8ff75d640 10 monclient: _renew_subs 2024-09-13T11:55:02.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8ff75d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:55:02.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8ff75d640 1 -- 172.21.15.100:0/2213267148 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8f811f5f0 con 0x7ff8f8137770 2024-09-13T11:55:02.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8f77fe640 4 set_mon_vals no callback set 2024-09-13T11:55:02.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8ff75d640 10 monclient: _renew_subs 2024-09-13T11:55:02.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8ff75d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:55:02.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.277+0000 7ff8ff75d640 1 -- 172.21.15.100:0/2213267148 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8f813e320 con 0x7ff8f8137770 2024-09-13T11:55:02.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.278+0000 7ff8ff75d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:02.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.278+0000 7ff8ff75d640 1 --2- 172.21.15.100:0/2213267148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8f81cc4f0 0x7ff8f81cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.278+0000 7ff8ff75d640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:02.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.278+0000 7ff8ff75d640 1 -- 172.21.15.100:0/2213267148 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7ff8f80a3230 con 0x7ff8f81cc4f0 2024-09-13T11:55:02.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.278+0000 7ff8f5ffb640 1 -- 172.21.15.100:0/2213267148 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7ff8e8047460 con 0x7ff8f8137770 2024-09-13T11:55:02.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.278+0000 7ff8fd4f9640 1 --2- 172.21.15.100:0/2213267148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8f81cc4f0 0x7ff8f81cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:02.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.278+0000 7ff8fd4f9640 10 monclient: get_auth_request con 0x7ff8f81cc4f0 auth_method 0 2024-09-13T11:55:02.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.278+0000 7ff8fd4f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:02.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.278+0000 7ff8fd4f9640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:02.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.278+0000 7ff8fd4f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:02.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.278+0000 7ff8fd4f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:02.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.278+0000 7ff8fd4f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:02.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.278+0000 7ff8fd4f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:02.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.279+0000 7ff8fd4f9640 10 monclient(hunting): handle_auth_done global_id 4889 payload 824 2024-09-13T11:55:02.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.279+0000 7ff8fd4f9640 1 -- 172.21.15.100:0/2213267148 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff8f811fa00 con 0x7ff8f81cc4f0 2024-09-13T11:55:02.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.279+0000 7ff8f5ffb640 1 --2- 172.21.15.100:0/2213267148 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff8e403dfa0 0x7ff8e4040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.279+0000 7ff8f5ffb640 1 -- 172.21.15.100:0/2213267148 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(115..115 src has 1..115) ==== 4346+0+0 (secure 0 0 0) 0x7ff8e8072fc0 con 0x7ff8f8137770 2024-09-13T11:55:02.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.279+0000 7ff8fd4f9640 1 --2- 172.21.15.100:0/2213267148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8f81cc4f0 0x7ff8f81cc8d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff8f000f770 tx=0x7ff8f000fc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:02.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.279+0000 7ff8fccf8640 1 --2- 172.21.15.100:0/2213267148 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff8e403dfa0 0x7ff8e4040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:02.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.279+0000 7ff8fccf8640 10 monclient: get_auth_request con 0x7ff8e403dfa0 auth_method 0 2024-09-13T11:55:02.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.279+0000 7ff8f5ffb640 1 -- 172.21.15.100:0/2213267148 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f00050d0 con 0x7ff8f81cc4f0 2024-09-13T11:55:02.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.279+0000 7ff8f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:02.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.279+0000 7ff8f5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:55:02.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.279+0000 7ff8f5ffb640 10 monclient: dump: 2024-09-13T11:55:02.305 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:02.305 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:02.305 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.306 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.306 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:02.306 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.306 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:02.306 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:02.306 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:02.306 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.306 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.279+0000 7ff8f5ffb640 1 -- 172.21.15.100:0/2213267148 <== mon.2 v2:172.21.15.100:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff8f0018070 con 0x7ff8f81cc4f0 2024-09-13T11:55:02.306 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.279+0000 7ff8f5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:02.306 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.279+0000 7ff8f5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:55:02.306 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.279+0000 7ff8f5ffb640 1 -- 172.21.15.100:0/2213267148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8f81cc4f0 msgr2=0x7ff8f81cc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.306 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.279+0000 7ff8f5ffb640 1 --2- 172.21.15.100:0/2213267148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8f81cc4f0 0x7ff8f81cc8d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff8f000f770 tx=0x7ff8f000fc30 comp rx=0 tx=0).stop 2024-09-13T11:55:02.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.279+0000 7ff8f5ffb640 1 -- 172.21.15.100:0/2213267148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8f81cc4f0 msgr2=0x7ff8f81cc8d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:55:02.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.279+0000 7ff8f5ffb640 1 --2- 172.21.15.100:0/2213267148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8f81cc4f0 0x7ff8f81cc8d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.279+0000 7ff8fccf8640 1 --2- 172.21.15.100:0/2213267148 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff8e403dfa0 0x7ff8e4040460 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec007e50 tx=0x7ff8ec007da0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:02.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.286+0000 7ff8ff75d640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-13T11:55:02.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.286+0000 7ff8ff75d640 1 --2- 172.21.15.100:0/2213267148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8f81cce10 0x7ff8f813a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.286+0000 7ff8ff75d640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:02.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.286+0000 7ff8ff75d640 1 -- 172.21.15.100:0/2213267148 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7ff8f8137b50 con 0x7ff8f81cce10 2024-09-13T11:55:02.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.286+0000 7ff8fd4f9640 1 --2- 172.21.15.100:0/2213267148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8f81cce10 0x7ff8f813a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:02.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.286+0000 7ff8fd4f9640 10 monclient: get_auth_request con 0x7ff8f81cce10 auth_method 0 2024-09-13T11:55:02.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.286+0000 7ff8fd4f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:02.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.286+0000 7ff8fd4f9640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:02.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.286+0000 7ff8fd4f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:02.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.286+0000 7ff8fd4f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:02.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.286+0000 7ff8fd4f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:02.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.286+0000 7ff8fd4f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:02.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.286+0000 7ff8fd4f9640 10 monclient(hunting): handle_auth_done global_id 4892 payload 824 2024-09-13T11:55:02.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.286+0000 7ff8fd4f9640 1 -- 172.21.15.100:0/2213267148 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7ff8f813f5c0 con 0x7ff8f81cce10 2024-09-13T11:55:02.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.287+0000 7ff8fd4f9640 1 --2- 172.21.15.100:0/2213267148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8f81cce10 0x7ff8f813a4b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0017000 tx=0x7ff8f0001120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:02.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.287+0000 7ff8f5ffb640 1 -- 172.21.15.100:0/2213267148 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f002c080 con 0x7ff8f81cce10 2024-09-13T11:55:02.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.287+0000 7ff8f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:02.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.287+0000 7ff8f5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:55:02.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.287+0000 7ff8f5ffb640 10 monclient: dump: 2024-09-13T11:55:02.310 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:02.310 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:02.310 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.310 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.311 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:02.311 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.311 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:02.311 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:02.311 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:02.311 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.311 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.287+0000 7ff8f5ffb640 1 -- 172.21.15.100:0/2213267148 <== mon.2 v2:172.21.15.100:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7ff8f000d9d0 con 0x7ff8f81cce10 2024-09-13T11:55:02.311 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.287+0000 7ff8f5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-13T11:55:02.311 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.287+0000 7ff8f5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:55:02.311 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.287+0000 7ff8f5ffb640 1 -- 172.21.15.100:0/2213267148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8f81cce10 msgr2=0x7ff8f813a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.311 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.287+0000 7ff8f5ffb640 1 --2- 172.21.15.100:0/2213267148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8f81cce10 0x7ff8f813a4b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0017000 tx=0x7ff8f0001120 comp rx=0 tx=0).stop 2024-09-13T11:55:02.311 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.287+0000 7ff8f5ffb640 1 -- 172.21.15.100:0/2213267148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8f81cce10 msgr2=0x7ff8f813a4b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:55:02.312 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.287+0000 7ff8f5ffb640 1 --2- 172.21.15.100:0/2213267148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8f81cce10 0x7ff8f813a4b0 secure :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0017000 tx=0x7ff8f0001120 comp rx=0 tx=0).stop 2024-09-13T11:55:02.312 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:55:02.312 INFO:teuthology.orchestra.run.smithi100.stdout: "ret": 0, 2024-09-13T11:55:02.312 INFO:teuthology.orchestra.run.smithi100.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-13T11:55:02.312 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:55:02.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.287+0000 7ff8ff75d640 1 -- 172.21.15.100:0/2213267148 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff8e403dfa0 msgr2=0x7ff8e4040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.287+0000 7ff8ff75d640 1 --2- 172.21.15.100:0/2213267148 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff8e403dfa0 0x7ff8e4040460 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec007e50 tx=0x7ff8ec007da0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.287+0000 7ff8ff75d640 10 monclient: shutdown 2024-09-13T11:55:02.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.287+0000 7ff8ff75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:55:02.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.287+0000 7ff8ff75d640 1 -- 172.21.15.100:0/2213267148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8f8137770 msgr2=0x7ff8f811dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.287+0000 7ff8ff75d640 1 --2- 172.21.15.100:0/2213267148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8f8137770 0x7ff8f811dd00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8031000 tx=0x7ff8e80046c0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.287+0000 7ff8fdcfa640 1 -- 172.21.15.100:0/2213267148 reap_dead start 2024-09-13T11:55:02.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.288+0000 7ff8fdcfa640 1 -- 172.21.15.100:0/2213267148 reap_dead start 2024-09-13T11:55:02.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.288+0000 7ff8ff75d640 1 -- 172.21.15.100:0/2213267148 shutdown_connections 2024-09-13T11:55:02.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.288+0000 7ff8ff75d640 1 -- 172.21.15.100:0/2213267148 >> 172.21.15.100:0/2213267148 conn(0x7ff8f80a7260 msgr2=0x7ff8f80a1530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:55:02.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.288+0000 7ff8ff75d640 1 -- 172.21.15.100:0/2213267148 shutdown_connections 2024-09-13T11:55:02.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.288+0000 7ff8ff75d640 1 -- 172.21.15.100:0/2213267148 wait complete. 2024-09-13T11:55:02.315 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-13T11:55:02.315 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-13T11:55:02.315 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:55:02.326 INFO:tasks.ceph.mon.c:Stopped 2024-09-13T11:55:02.326 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-13T11:55:02.326 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:55:02.461 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.460+0000 7fad2a42b640 10 monclient: get_monmap_and_config 2024-09-13T11:55:02.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.460+0000 7fad2a42b640 10 monclient: build_initial_monmap 2024-09-13T11:55:02.462 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.460+0000 7fad2a42b640 10 monclient: monmap: 2024-09-13T11:55:02.462 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:55:02.462 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:02.462 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:55:02.461383+0000 2024-09-13T11:55:02.462 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:55:02.461383+0000 2024-09-13T11:55:02.462 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:55:02.462 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.462 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:55:02.462 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:55:02.462 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:55:02.462 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:55:02.463 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:55:02.463 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.461+0000 7fad2a42b640 1 Processor -- start 2024-09-13T11:55:02.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.461+0000 7fad2a42b640 1 -- start start 2024-09-13T11:55:02.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.461+0000 7fad2a42b640 10 monclient: init 2024-09-13T11:55:02.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.461+0000 7fad2a42b640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:02.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.461+0000 7fad2a42b640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-13T11:55:02.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.461+0000 7fad2a42b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad240a2640 0x7fad240a2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.461+0000 7fad2a42b640 10 monclient(hunting): picked mon.noname-b con 0x7fad240a2640 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:55:02.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.461+0000 7fad2a42b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fad240a2f60 addr v1:172.21.15.100:6789/0 2024-09-13T11:55:02.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.461+0000 7fad2a42b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fad24135b10 0x7fad2413d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.461+0000 7fad2a42b640 10 monclient(hunting): picked mon.noname-c con 0x7fad24135b10 addr v2:172.21.15.122:3300/0 2024-09-13T11:55:02.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.461+0000 7fad2a42b640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad24076830 con 0x7fad240a2f60 2024-09-13T11:55:02.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.461+0000 7fad2a42b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:02.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.461+0000 7fad2a42b640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7fad24076f50 con 0x7fad24135b10 2024-09-13T11:55:02.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.461+0000 7fad2a42b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:02.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.461+0000 7fad2a42b640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fad2409f040 con 0x7fad240a2640 2024-09-13T11:55:02.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.461+0000 7fad2a42b640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:02.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.461+0000 7fad23fff640 1 -- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad240a2640 msgr2=0x7fad240a2a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.100:3301/0 2024-09-13T11:55:02.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.461+0000 7fad2a42b640 10 monclient(hunting): authenticate will time out at 883.054960s 2024-09-13T11:55:02.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.461+0000 7fad23fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad240a2640 0x7fad240a2a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:55:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad237fe640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fad240a2f60 0x7fad24135400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:35552/0 (socket says 172.21.15.100:35552) 2024-09-13T11:55:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad237fe640 1 -- 172.21.15.100:0/4251728165 learned_addr learned my addr 172.21.15.100:0/4251728165 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:55:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad289c8640 1 --2- 172.21.15.100:0/4251728165 >> v2:172.21.15.122:3300/0 conn(0x7fad24135b10 0x7fad2413d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 1 -- 172.21.15.100:0/4251728165 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1847431353 0 0) 0x7fad24076830 con 0x7fad240a2f60 2024-09-13T11:55:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 10 monclient(hunting): _negotiate 2024-09-13T11:55:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 10 monclient(hunting): my global_id is 14199 2024-09-13T11:55:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 1 -- 172.21.15.100:0/4251728165 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad0c003520 con 0x7fad240a2f60 2024-09-13T11:55:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad289c8640 10 monclient(hunting): get_auth_request con 0x7fad24135b10 auth_method 0 2024-09-13T11:55:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad289c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad289c8640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad289c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad289c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad289c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad289c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 1 -- 172.21.15.100:0/4251728165 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 637311402 0 0) 0x7fad0c003520 con 0x7fad240a2f60 2024-09-13T11:55:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 1 -- 172.21.15.100:0/4251728165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad240a2640 msgr2=0x7fad240a2a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:55:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 1 --2- 172.21.15.100:0/4251728165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad240a2640 0x7fad240a2a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 1 -- 172.21.15.100:0/4251728165 >> v2:172.21.15.122:3300/0 conn(0x7fad24135b10 msgr2=0x7fad2413d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:55:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 1 --2- 172.21.15.100:0/4251728165 >> v2:172.21.15.122:3300/0 conn(0x7fad24135b10 0x7fad2413d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 10 monclient: _finish_hunting 0 2024-09-13T11:55:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 1 monclient: found mon.noname-a-legacy 2024-09-13T11:55:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.100:6789/0 2024-09-13T11:55:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 1 -- 172.21.15.100:0/4251728165 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad240765b0 con 0x7fad240a2f60 2024-09-13T11:55:02.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 10 monclient: _finish_auth 0 2024-09-13T11:55:02.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:55:02.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:02.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad2a42b640 5 monclient: authenticate success, global_id 14199 2024-09-13T11:55:02.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad2a42b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:55:02.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 1 -- 172.21.15.100:0/4251728165 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 626700363 0 0) 0x7fad180016f0 con 0x7fad240a2f60 2024-09-13T11:55:02.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:02.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-13T11:55:02.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 10 monclient: dump: 2024-09-13T11:55:02.476 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:02.477 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:02.477 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.477 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.477 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:02.477 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.477 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:02.477 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:02.477 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:02.477 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.477 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] but i'm connected to v1:172.21.15.100:6789/0, reconnecting 2024-09-13T11:55:02.477 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:02.477 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 1 -- 172.21.15.100:0/4251728165 >> v1:172.21.15.100:6789/0 conn(0x7fad240a2f60 legacy=0x7fad24135400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.478 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:55:02.478 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 1 --2- 172.21.15.100:0/4251728165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad240a2640 0x7fad0c0039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.478 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 10 monclient(hunting): picked mon.c con 0x7fad240a2640 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:55:02.478 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 1 --2- 172.21.15.100:0/4251728165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad0c003fa0 0x7fad0c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.478 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 10 monclient(hunting): picked mon.b con 0x7fad0c003fa0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:55:02.478 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 1 --2- 172.21.15.100:0/4251728165 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad0c004d40 0x7fad0c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.478 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 10 monclient(hunting): picked mon.a con 0x7fad0c004d40 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:55:02.478 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:02.478 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 1 -- 172.21.15.100:0/4251728165 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fad2409f040 con 0x7fad0c004d40 2024-09-13T11:55:02.479 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:02.479 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 1 -- 172.21.15.100:0/4251728165 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fad0c00bc20 con 0x7fad0c003fa0 2024-09-13T11:55:02.479 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:02.479 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 1 -- 172.21.15.100:0/4251728165 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fad0c00be20 con 0x7fad240a2640 2024-09-13T11:55:02.479 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:02.479 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad22ffd640 10 monclient(hunting): _finish_auth 0 2024-09-13T11:55:02.479 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad23fff640 1 -- 172.21.15.100:0/4251728165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad240a2640 msgr2=0x7fad0c0039d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.100:3301/0 2024-09-13T11:55:02.479 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad2a42b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-13T11:55:02.479 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad23fff640 1 --2- 172.21.15.100:0/4251728165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad240a2640 0x7fad0c0039d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:55:02.479 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad289c8640 1 --2- 172.21.15.100:0/4251728165 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad0c004d40 0x7fad0c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:02.479 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad289c8640 10 monclient(hunting): get_auth_request con 0x7fad0c004d40 auth_method 0 2024-09-13T11:55:02.479 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad289c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:02.479 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad289c8640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:02.480 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad289c8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:55:02.480 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad237fe640 1 --2- 172.21.15.100:0/4251728165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad0c003fa0 0x7fad0c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:02.480 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad237fe640 10 monclient(hunting): get_auth_request con 0x7fad0c003fa0 auth_method 0 2024-09-13T11:55:02.480 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:02.480 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad237fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:02.480 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad237fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:55:02.480 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad289c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:02.480 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad289c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:02.480 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad289c8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:55:02.480 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:02.480 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:02.480 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad237fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:55:02.481 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad289c8640 10 monclient(hunting): handle_auth_done global_id 14199 payload 293 2024-09-13T11:55:02.481 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad289c8640 1 -- 172.21.15.100:0/4251728165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad240a2640 msgr2=0x7fad0c0039d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:55:02.481 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad289c8640 1 --2- 172.21.15.100:0/4251728165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad240a2640 0x7fad0c0039d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.481 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad289c8640 1 -- 172.21.15.100:0/4251728165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad0c003fa0 msgr2=0x7fad0c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:55:02.481 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad289c8640 1 --2- 172.21.15.100:0/4251728165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad0c003fa0 0x7fad0c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.481 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad289c8640 10 monclient: _finish_hunting 0 2024-09-13T11:55:02.481 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.462+0000 7fad289c8640 1 monclient: found mon.a 2024-09-13T11:55:02.481 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.463+0000 7fad289c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:02.481 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.463+0000 7fad289c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:02.481 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.463+0000 7fad289c8640 1 -- 172.21.15.100:0/4251728165 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad0c003520 con 0x7fad0c004d40 2024-09-13T11:55:02.481 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.463+0000 7fad289c8640 10 monclient: _finish_auth 0 2024-09-13T11:55:02.482 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.463+0000 7fad289c8640 10 monclient: _check_auth_tickets 2024-09-13T11:55:02.482 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.463+0000 7fad289c8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:02.482 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.463+0000 7fad289c8640 1 --2- 172.21.15.100:0/4251728165 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad0c004d40 0x7fad0c0051c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fad1c009fc0 tx=0x7fad0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b9701e09fb2447 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:02.482 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.463+0000 7fad22ffd640 1 -- 172.21.15.100:0/4251728165 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c009c10 con 0x7fad0c004d40 2024-09-13T11:55:02.482 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.463+0000 7fad22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:02.482 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.463+0000 7fad22ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:02.482 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.463+0000 7fad22ffd640 10 monclient: dump: 2024-09-13T11:55:02.482 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:02.482 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:02.482 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.483 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.483 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:02.483 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.483 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:02.483 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:02.483 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:02.483 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.483 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.463+0000 7fad22ffd640 1 -- 172.21.15.100:0/4251728165 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad1c036070 con 0x7fad0c004d40 2024-09-13T11:55:02.483 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.463+0000 7fad2a42b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:55:02.483 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.463+0000 7fad22ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:55:02.483 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.463+0000 7fad2a42b640 10 monclient: get_monmap_and_config success 2024-09-13T11:55:02.483 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.463+0000 7fad22ffd640 1 -- 172.21.15.100:0/4251728165 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c02d070 con 0x7fad0c004d40 2024-09-13T11:55:02.483 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.463+0000 7fad22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:02.484 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.463+0000 7fad2a42b640 4 set_mon_vals no callback set 2024-09-13T11:55:02.484 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.463+0000 7fad22ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:02.484 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.463+0000 7fad22ffd640 10 monclient: dump: 2024-09-13T11:55:02.484 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:02.484 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:02.484 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.484 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.484 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:02.484 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.484 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:02.484 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:02.484 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:02.484 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.485 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.464+0000 7fad2a42b640 10 monclient: shutdown 2024-09-13T11:55:02.485 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.464+0000 7fad2a42b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:55:02.485 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.464+0000 7fad2a42b640 1 -- 172.21.15.100:0/4251728165 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad0c004d40 msgr2=0x7fad0c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.485 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.464+0000 7fad2a42b640 1 --2- 172.21.15.100:0/4251728165 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad0c004d40 0x7fad0c0051c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fad1c009fc0 tx=0x7fad0c005830 comp rx=0 tx=0).stop 2024-09-13T11:55:02.485 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.464+0000 7fad289c8640 1 -- 172.21.15.100:0/4251728165 reap_dead start 2024-09-13T11:55:02.485 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.464+0000 7fad2a42b640 1 -- 172.21.15.100:0/4251728165 shutdown_connections 2024-09-13T11:55:02.485 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.464+0000 7fad2a42b640 1 -- 172.21.15.100:0/4251728165 >> 172.21.15.100:0/4251728165 conn(0x7fad240a7260 msgr2=0x7fad240a3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:55:02.485 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.464+0000 7fad2a42b640 1 -- 172.21.15.100:0/4251728165 shutdown_connections 2024-09-13T11:55:02.485 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.464+0000 7fad2a42b640 1 -- 172.21.15.100:0/4251728165 wait complete. 2024-09-13T11:55:02.485 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.465+0000 7fad2a42b640 10 monclient: build_initial_monmap 2024-09-13T11:55:02.485 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.465+0000 7fad2a42b640 10 monclient: monmap: 2024-09-13T11:55:02.485 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:55:02.485 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:55:02.486 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:55:02.486 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:55:02.486 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:55:02.486 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.486 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:55:02.486 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:55:02.486 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:55:02.486 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.486 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.465+0000 7fad2a42b640 1 Processor -- start 2024-09-13T11:55:02.486 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.465+0000 7fad2a42b640 1 -- start start 2024-09-13T11:55:02.486 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.465+0000 7fad2a42b640 10 monclient: init 2024-09-13T11:55:02.486 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.465+0000 7fad2a42b640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:02.486 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.465+0000 7fad2a42b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:55:02.486 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.465+0000 7fad2a42b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad0c004d40 0x7fad24121a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.487 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.465+0000 7fad2a42b640 10 monclient(hunting): picked mon.noname-c con 0x7fad0c004d40 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:55:02.487 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.465+0000 7fad2a42b640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad24121f70 0x7fad241223d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.487 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.465+0000 7fad2a42b640 10 monclient(hunting): picked mon.noname-a con 0x7fad24121f70 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:55:02.487 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.465+0000 7fad2a42b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad241ca630 0x7fad241caae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.487 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.465+0000 7fad2a42b640 10 monclient(hunting): picked mon.noname-b con 0x7fad241ca630 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:55:02.487 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.465+0000 7fad2a42b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:02.487 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.465+0000 7fad2a42b640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fad240a3340 con 0x7fad24121f70 2024-09-13T11:55:02.487 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.465+0000 7fad2a42b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:02.487 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.465+0000 7fad2a42b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fad240a0fe0 con 0x7fad241ca630 2024-09-13T11:55:02.487 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.465+0000 7fad2a42b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:02.487 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.465+0000 7fad2a42b640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fad241345a0 con 0x7fad0c004d40 2024-09-13T11:55:02.487 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.465+0000 7fad2a42b640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:02.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.465+0000 7fad23fff640 1 -- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad0c004d40 msgr2=0x7fad24121a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:3301/0 2024-09-13T11:55:02.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.465+0000 7fad2a42b640 10 monclient(hunting): authenticate will time out at 883.058956s 2024-09-13T11:55:02.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.465+0000 7fad23fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad0c004d40 0x7fad24121a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:55:02.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad237fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad24121f70 0x7fad241223d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:02.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad237fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad24121f70 0x7fad241223d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47042/0 (socket says 172.21.15.100:47042) 2024-09-13T11:55:02.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad237fe640 1 -- 172.21.15.100:0/2004157444 learned_addr learned my addr 172.21.15.100:0/2004157444 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:55:02.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad237fe640 10 monclient(hunting): get_auth_request con 0x7fad24121f70 auth_method 0 2024-09-13T11:55:02.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:02.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad237fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:02.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:02.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:02.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:02.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:02.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad289c8640 1 --2- 172.21.15.100:0/2004157444 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad241ca630 0x7fad241caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:02.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad237fe640 10 monclient(hunting): handle_auth_done global_id 14202 payload 824 2024-09-13T11:55:02.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad237fe640 1 -- 172.21.15.100:0/2004157444 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad0c004d40 msgr2=0x7fad24121a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:55:02.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad237fe640 1 --2- 172.21.15.100:0/2004157444 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad0c004d40 0x7fad24121a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad237fe640 1 -- 172.21.15.100:0/2004157444 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad241ca630 msgr2=0x7fad241caae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad237fe640 1 --2- 172.21.15.100:0/2004157444 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad241ca630 0x7fad241caae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad237fe640 10 monclient: _finish_hunting 0 2024-09-13T11:55:02.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad237fe640 1 monclient: found mon.noname-a 2024-09-13T11:55:02.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:02.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:55:02.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad237fe640 1 -- 172.21.15.100:0/2004157444 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad241249e0 con 0x7fad24121f70 2024-09-13T11:55:02.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad289c8640 10 monclient: get_auth_request con 0x7fad241ca630 auth_method 0 2024-09-13T11:55:02.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad289c8640 1 --2- 172.21.15.100:0/2004157444 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad241ca630 0x7fad241caae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:55:02.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad237fe640 1 --2- 172.21.15.100:0/2004157444 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad24121f70 0x7fad241223d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fad18007fc0 tx=0x7fad180074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:02.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad20ff9640 1 -- 172.21.15.100:0/2004157444 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1801a070 con 0x7fad24121f70 2024-09-13T11:55:02.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:02.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad20ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:55:02.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad20ff9640 10 monclient: dump: 2024-09-13T11:55:02.490 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:02.490 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:02.490 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.490 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.490 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:02.491 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.491 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:02.491 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:02.491 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:02.491 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad20ff9640 10 monclient: _finish_auth 0 2024-09-13T11:55:02.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad20ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:55:02.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:02.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad2a42b640 5 monclient: authenticate success, global_id 14202 2024-09-13T11:55:02.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad20ff9640 1 -- 172.21.15.100:0/2004157444 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad18007740 con 0x7fad24121f70 2024-09-13T11:55:02.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad20ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:55:02.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad20ff9640 1 -- 172.21.15.100:0/2004157444 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1800fc00 con 0x7fad24121f70 2024-09-13T11:55:02.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:02.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad20ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:02.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad20ff9640 10 monclient: dump: 2024-09-13T11:55:02.492 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:02.492 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:02.492 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.492 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:02.492 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:02.492 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:02.492 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:02.492 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:02.492 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:02.492 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:02.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad2a42b640 10 monclient: _renew_subs 2024-09-13T11:55:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad227fc640 4 set_mon_vals no callback set 2024-09-13T11:55:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad2a42b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad2a42b640 1 -- 172.21.15.100:0/2004157444 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad24125c70 con 0x7fad24121f70 2024-09-13T11:55:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad2a42b640 10 monclient: _renew_subs 2024-09-13T11:55:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad2a42b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.466+0000 7fad2a42b640 1 -- 172.21.15.100:0/2004157444 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad24074f90 con 0x7fad24121f70 2024-09-13T11:55:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.467+0000 7fad20ff9640 1 -- 172.21.15.100:0/2004157444 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fad1801d360 con 0x7fad24121f70 2024-09-13T11:55:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.467+0000 7fad2a42b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.467+0000 7fad2a42b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.467+0000 7fad2a42b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.467+0000 7fad2a42b640 1 -- 172.21.15.100:0/2004157444 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad24076830 con 0x7fad24121f70 2024-09-13T11:55:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.467+0000 7fad20ff9640 1 --2- 172.21.15.100:0/2004157444 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fad1003df50 0x7fad10040410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.467+0000 7fad20ff9640 1 -- 172.21.15.100:0/2004157444 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4346+0+0 (secure 0 0 0) 0x7fad18051d80 con 0x7fad24121f70 2024-09-13T11:55:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.467+0000 7fad23fff640 1 --2- 172.21.15.100:0/2004157444 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fad1003df50 0x7fad10040410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:02.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.467+0000 7fad23fff640 10 monclient: get_auth_request con 0x7fad1003df50 auth_method 0 2024-09-13T11:55:02.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.468+0000 7fad23fff640 1 --2- 172.21.15.100:0/2004157444 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fad1003df50 0x7fad10040410 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fad14003980 tx=0x7fad14006d90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:02.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.472+0000 7fad20ff9640 1 -- 172.21.15.100:0/2004157444 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fad18055030 con 0x7fad24121f70 2024-09-13T11:55:02.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.472+0000 7fad20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:02.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.472+0000 7fad20ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:55:02.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.714+0000 7fad2a42b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:55:02.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.714+0000 7fad2a42b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:55:02.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.714+0000 7fad2a42b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:02.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.714+0000 7fad2a42b640 1 -- 172.21.15.100:0/2004157444 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fad240771e0 con 0x7fad24121f70 2024-09-13T11:55:02.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.714+0000 7fad20ff9640 1 -- 172.21.15.100:0/2004157444 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fad24125c70 con 0x7fad24121f70 2024-09-13T11:55:02.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.714+0000 7fad20ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:55:02.716 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.714+0000 7fad20ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:55:02.716 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:55:02.718 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.716+0000 7fad2a42b640 1 -- 172.21.15.100:0/2004157444 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fad1003df50 msgr2=0x7fad10040410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.718 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.716+0000 7fad2a42b640 1 --2- 172.21.15.100:0/2004157444 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fad1003df50 0x7fad10040410 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fad14003980 tx=0x7fad14006d90 comp rx=0 tx=0).stop 2024-09-13T11:55:02.718 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.716+0000 7fad2a42b640 10 monclient: shutdown 2024-09-13T11:55:02.718 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.716+0000 7fad2a42b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:55:02.718 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.716+0000 7fad2a42b640 1 -- 172.21.15.100:0/2004157444 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad24121f70 msgr2=0x7fad241223d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:02.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.716+0000 7fad2a42b640 1 --2- 172.21.15.100:0/2004157444 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad24121f70 0x7fad241223d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fad18007fc0 tx=0x7fad180074f0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.716+0000 7fad2a42b640 1 -- 172.21.15.100:0/2004157444 shutdown_connections 2024-09-13T11:55:02.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.716+0000 7fad2a42b640 1 --2- 172.21.15.100:0/2004157444 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fad1003df50 0x7fad10040410 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.716+0000 7fad2a42b640 1 --2- 172.21.15.100:0/2004157444 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad24121f70 0x7fad241223d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.716+0000 7fad2a42b640 1 --2- 172.21.15.100:0/2004157444 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad241ca630 0x7fad241caae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.716+0000 7fad2a42b640 1 --2- 172.21.15.100:0/2004157444 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad0c004d40 0x7fad24121a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:02.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.716+0000 7fad2a42b640 1 -- 172.21.15.100:0/2004157444 >> 172.21.15.100:0/2004157444 conn(0x7fad240a7260 msgr2=0x7fad24071e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:55:02.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.716+0000 7fad2a42b640 1 -- 172.21.15.100:0/2004157444 shutdown_connections 2024-09-13T11:55:02.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:02.716+0000 7fad2a42b640 1 -- 172.21.15.100:0/2004157444 wait complete. 2024-09-13T11:55:05.731 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:55:05.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.838+0000 7f1fb13ec640 10 monclient: get_monmap_and_config 2024-09-13T11:55:05.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.838+0000 7f1fb13ec640 10 monclient: build_initial_monmap 2024-09-13T11:55:05.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.838+0000 7f1fb13ec640 10 monclient: monmap: 2024-09-13T11:55:05.840 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:55:05.840 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:05.840 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:55:05.839197+0000 2024-09-13T11:55:05.840 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:55:05.839197+0000 2024-09-13T11:55:05.840 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:55:05.840 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:05.840 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:55:05.840 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:55:05.840 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:55:05.840 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:55:05.840 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:55:05.841 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:05.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.838+0000 7f1fb13ec640 1 Processor -- start 2024-09-13T11:55:05.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1fb13ec640 1 -- start start 2024-09-13T11:55:05.848 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1fb13ec640 10 monclient: init 2024-09-13T11:55:05.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1fb13ec640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:05.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1fb13ec640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-13T11:55:05.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1fb13ec640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1fac137a60 0x7f1fac0a2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:05.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1fb13ec640 10 monclient(hunting): picked mon.noname-a con 0x7f1fac137a60 addr v2:172.21.15.100:3300/0 2024-09-13T11:55:05.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1fb13ec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1fac137e40 addr v1:172.21.15.100:6789/0 2024-09-13T11:55:05.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1fb13ec640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1fac138250 0x7f1fac0a5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:05.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1fb13ec640 10 monclient(hunting): picked mon.noname-c con 0x7f1fac138250 addr v2:172.21.15.122:3300/0 2024-09-13T11:55:05.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1fb13ec640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fac0a5be0 con 0x7f1fac137e40 2024-09-13T11:55:05.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1fb13ec640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:05.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1fb13ec640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f1fac0a4410 con 0x7f1fac137a60 2024-09-13T11:55:05.849 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1fb13ec640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:05.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1fb13ec640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f1fac0a76b0 con 0x7f1fac138250 2024-09-13T11:55:05.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1fb13ec640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:05.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1fb13ec640 10 monclient(hunting): authenticate will time out at 886.432591s 2024-09-13T11:55:05.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1faaffd640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1fac137a60 0x7f1fac0a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:05.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1fa27fc640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f1fac137e40 0x7f1fac0a4950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:35558/0 (socket says 172.21.15.100:35558) 2024-09-13T11:55:05.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1fa27fc640 1 -- 172.21.15.100:0/2479340619 learned_addr learned my addr 172.21.15.100:0/2479340619 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:55:05.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1faaffd640 10 monclient(hunting): get_auth_request con 0x7f1fac137a60 auth_method 0 2024-09-13T11:55:05.850 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1faaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:05.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1faaffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:05.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1faaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:05.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1faaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:05.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1faaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:05.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1faaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:05.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.839+0000 7f1fab7fe640 1 --2- 172.21.15.100:0/2479340619 >> v2:172.21.15.122:3300/0 conn(0x7f1fac138250 0x7f1fac0a5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:05.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1faaffd640 10 monclient(hunting): handle_auth_done global_id 14205 payload 274 2024-09-13T11:55:05.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1faaffd640 1 -- 172.21.15.100:0/2479340619 >> v2:172.21.15.122:3300/0 conn(0x7f1fac138250 msgr2=0x7f1fac0a5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:55:05.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1faaffd640 1 --2- 172.21.15.100:0/2479340619 >> v2:172.21.15.122:3300/0 conn(0x7f1fac138250 0x7f1fac0a5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:05.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1faaffd640 1 -- 172.21.15.100:0/2479340619 >> v1:172.21.15.100:6789/0 conn(0x7f1fac137e40 legacy=0x7f1fac0a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:05.851 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1faaffd640 10 monclient: _finish_hunting 0 2024-09-13T11:55:05.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1faaffd640 1 monclient: found mon.noname-a 2024-09-13T11:55:05.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1faaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:05.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1faaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:55:05.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1faaffd640 1 -- 172.21.15.100:0/2479340619 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fac0a19e0 con 0x7f1fac137a60 2024-09-13T11:55:05.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1fab7fe640 10 monclient: get_auth_request con 0x7f1fac138250 auth_method 0 2024-09-13T11:55:05.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1fab7fe640 1 --2- 172.21.15.100:0/2479340619 >> v2:172.21.15.122:3300/0 conn(0x7f1fac138250 0x7f1fac0a5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:55:05.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1faaffd640 1 --2- 172.21.15.100:0/2479340619 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fac137a60 0x7f1fac0a2980 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1fac0a5db0 tx=0x7f1f9c02a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c02d5fa545126a4f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:05.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1faa7fc640 1 -- 172.21.15.100:0/2479340619 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c037070 con 0x7f1fac137a60 2024-09-13T11:55:05.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1faa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:05.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1faa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:55:05.852 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1faa7fc640 10 monclient: dump: 2024-09-13T11:55:05.852 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:05.853 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:05.853 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:05.853 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:05.853 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:05.853 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:05.853 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:05.853 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:05.853 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:05.853 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:05.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1faa7fc640 10 monclient: _finish_auth 0 2024-09-13T11:55:05.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1faa7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:55:05.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1faa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:05.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1faa7fc640 1 -- 172.21.15.100:0/2479340619 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f9c02add0 con 0x7f1fac137a60 2024-09-13T11:55:05.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1fb13ec640 5 monclient: authenticate success, global_id 14205 2024-09-13T11:55:05.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1fb13ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:55:05.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1faa7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:55:05.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1fb13ec640 10 monclient: get_monmap_and_config success 2024-09-13T11:55:05.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1faa7fc640 1 -- 172.21.15.100:0/2479340619 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c032660 con 0x7f1fac137a60 2024-09-13T11:55:05.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1faa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:05.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1fb13ec640 4 set_mon_vals no callback set 2024-09-13T11:55:05.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1faa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:05.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.840+0000 7f1faa7fc640 10 monclient: dump: 2024-09-13T11:55:05.854 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:05.854 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:05.854 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:05.855 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:05.855 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:05.855 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:05.855 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:05.855 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:05.855 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:05.855 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:05.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.841+0000 7f1fb13ec640 10 monclient: shutdown 2024-09-13T11:55:05.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.841+0000 7f1fb13ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:55:05.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.841+0000 7f1fb13ec640 1 -- 172.21.15.100:0/2479340619 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fac137a60 msgr2=0x7f1fac0a2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:05.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.841+0000 7f1fb13ec640 1 --2- 172.21.15.100:0/2479340619 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fac137a60 0x7f1fac0a2980 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1fac0a5db0 tx=0x7f1f9c02a080 comp rx=0 tx=0).stop 2024-09-13T11:55:05.855 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.841+0000 7f1fb13ec640 1 -- 172.21.15.100:0/2479340619 shutdown_connections 2024-09-13T11:55:05.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.841+0000 7f1fb13ec640 1 --2- 172.21.15.100:0/2479340619 >> v2:172.21.15.122:3300/0 conn(0x7f1fac138250 0x7f1fac0a5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:05.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.841+0000 7f1fb13ec640 1 --2- 172.21.15.100:0/2479340619 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fac137a60 0x7f1fac0a2980 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:05.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.841+0000 7f1fb13ec640 1 -- 172.21.15.100:0/2479340619 >> 172.21.15.100:0/2479340619 conn(0x7f1fac0aa580 msgr2=0x7f1fac0a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:55:05.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.841+0000 7f1fb13ec640 1 -- 172.21.15.100:0/2479340619 shutdown_connections 2024-09-13T11:55:05.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.841+0000 7f1fb13ec640 1 -- 172.21.15.100:0/2479340619 wait complete. 2024-09-13T11:55:05.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.842+0000 7f1fb13ec640 10 monclient: build_initial_monmap 2024-09-13T11:55:05.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.842+0000 7f1fb13ec640 10 monclient: monmap: 2024-09-13T11:55:05.856 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:55:05.856 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:55:05.856 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:55:05.856 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:55:05.857 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:55:05.857 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:05.857 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:55:05.857 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:55:05.857 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:55:05.857 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:05.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.842+0000 7f1fb13ec640 1 Processor -- start 2024-09-13T11:55:05.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.842+0000 7f1fb13ec640 1 -- start start 2024-09-13T11:55:05.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.842+0000 7f1fb13ec640 10 monclient: init 2024-09-13T11:55:05.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fb13ec640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:05.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fb13ec640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:55:05.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fb13ec640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1fac137a60 0x7f1fac136590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:05.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fb13ec640 10 monclient(hunting): picked mon.noname-b con 0x7f1fac137a60 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:55:05.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fb13ec640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fac137e40 0x7f1fac136ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:05.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fb13ec640 10 monclient(hunting): picked mon.noname-c con 0x7f1fac137e40 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:55:05.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fb13ec640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fac138250 0x7f1fac1205a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:05.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fb13ec640 10 monclient(hunting): picked mon.noname-a con 0x7f1fac138250 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:55:05.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fb13ec640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:05.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fb13ec640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f1fac06ff00 con 0x7f1fac138250 2024-09-13T11:55:05.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fb13ec640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:05.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fb13ec640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1fac06fd80 con 0x7f1fac137a60 2024-09-13T11:55:05.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fb13ec640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:05.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fb13ec640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f1fac0a55a0 con 0x7f1fac137e40 2024-09-13T11:55:05.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fb13ec640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:05.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fa27fc640 1 -- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fac137e40 msgr2=0x7f1fac136ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:3301/0 2024-09-13T11:55:05.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fb13ec640 10 monclient(hunting): authenticate will time out at 886.436267s 2024-09-13T11:55:05.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fa27fc640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fac137e40 0x7f1fac136ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:55:05.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fab7fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fac138250 0x7f1fac1205a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:05.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fab7fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fac138250 0x7f1fac1205a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47054/0 (socket says 172.21.15.100:47054) 2024-09-13T11:55:05.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fab7fe640 1 -- 172.21.15.100:0/4115970857 learned_addr learned my addr 172.21.15.100:0/4115970857 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:55:05.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fab7fe640 10 monclient(hunting): get_auth_request con 0x7f1fac138250 auth_method 0 2024-09-13T11:55:05.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:05.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:05.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:05.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:05.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:05.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:05.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1faaffd640 1 --2- 172.21.15.100:0/4115970857 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1fac137a60 0x7f1fac136590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:05.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1faaffd640 10 monclient(hunting): get_auth_request con 0x7f1fac137a60 auth_method 0 2024-09-13T11:55:05.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1faaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:05.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1faaffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:05.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1faaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:05.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1faaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:05.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1faaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:05.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1faaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:05.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fab7fe640 10 monclient(hunting): handle_auth_done global_id 14211 payload 824 2024-09-13T11:55:05.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fab7fe640 1 -- 172.21.15.100:0/4115970857 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fac137e40 msgr2=0x7f1fac136ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:55:05.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fab7fe640 1 --2- 172.21.15.100:0/4115970857 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fac137e40 0x7f1fac136ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:05.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fab7fe640 1 -- 172.21.15.100:0/4115970857 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1fac137a60 msgr2=0x7f1fac136590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:05.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fab7fe640 1 --2- 172.21.15.100:0/4115970857 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1fac137a60 0x7f1fac136590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:05.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fab7fe640 10 monclient: _finish_hunting 0 2024-09-13T11:55:05.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fab7fe640 1 monclient: found mon.noname-a 2024-09-13T11:55:05.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:05.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fab7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:55:05.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fab7fe640 1 -- 172.21.15.100:0/4115970857 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fac123b80 con 0x7f1fac138250 2024-09-13T11:55:05.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fab7fe640 1 --2- 172.21.15.100:0/4115970857 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fac138250 0x7f1fac1205a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1fa400cef0 tx=0x7f1fa400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:05.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fa3fff640 1 -- 172.21.15.100:0/4115970857 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa401a070 con 0x7f1fac138250 2024-09-13T11:55:05.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fa3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:05.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fa3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:55:05.861 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fa3fff640 10 monclient: dump: 2024-09-13T11:55:05.862 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:05.862 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:05.862 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:05.862 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:05.862 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:05.862 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:05.862 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:05.862 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:05.862 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:05.862 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:05.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fa3fff640 10 monclient: _finish_auth 0 2024-09-13T11:55:05.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fa3fff640 10 monclient: _check_auth_tickets 2024-09-13T11:55:05.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fa3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:05.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fa3fff640 1 -- 172.21.15.100:0/4115970857 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fa400d860 con 0x7f1fac138250 2024-09-13T11:55:05.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fb13ec640 5 monclient: authenticate success, global_id 14211 2024-09-13T11:55:05.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fb13ec640 10 monclient: _renew_subs 2024-09-13T11:55:05.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fb13ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:05.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fb13ec640 1 -- 172.21.15.100:0/4115970857 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fac123e60 con 0x7f1fac138250 2024-09-13T11:55:05.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fa3fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:55:05.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fa3fff640 1 -- 172.21.15.100:0/4115970857 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa400dc50 con 0x7f1fac138250 2024-09-13T11:55:05.863 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fa3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:05.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fa3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:05.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fa3fff640 10 monclient: dump: 2024-09-13T11:55:05.864 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:05.864 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:05.864 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:05.864 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:05.864 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:05.864 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:05.865 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:05.865 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:05.865 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:05.865 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:05.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.843+0000 7f1fa9ffb640 4 set_mon_vals no callback set 2024-09-13T11:55:05.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.844+0000 7f1fb13ec640 10 monclient: _renew_subs 2024-09-13T11:55:05.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.844+0000 7f1fb13ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:05.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.844+0000 7f1fb13ec640 1 -- 172.21.15.100:0/4115970857 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fac124140 con 0x7f1fac138250 2024-09-13T11:55:05.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.844+0000 7f1fa3fff640 1 -- 172.21.15.100:0/4115970857 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f1fa401d070 con 0x7f1fac138250 2024-09-13T11:55:05.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.844+0000 7f1fb13ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:05.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.844+0000 7f1fb13ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:05.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.844+0000 7f1fb13ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:05.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.844+0000 7f1fb13ec640 1 -- 172.21.15.100:0/4115970857 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fac1370e0 con 0x7f1fac138250 2024-09-13T11:55:05.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.845+0000 7f1fa3fff640 1 --2- 172.21.15.100:0/4115970857 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f1f8c03df50 0x7f1f8c040410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:05.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.845+0000 7f1fa3fff640 1 -- 172.21.15.100:0/4115970857 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4346+0+0 (secure 0 0 0) 0x7f1fa4052f50 con 0x7f1fac138250 2024-09-13T11:55:05.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.845+0000 7f1faaffd640 1 --2- 172.21.15.100:0/4115970857 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f1f8c03df50 0x7f1f8c040410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:05.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.845+0000 7f1faaffd640 10 monclient: get_auth_request con 0x7f1f8c03df50 auth_method 0 2024-09-13T11:55:05.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.845+0000 7f1faaffd640 1 --2- 172.21.15.100:0/4115970857 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f1f8c03df50 0x7f1f8c040410 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c034000 tx=0x7f1f9c02e000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:05.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.849+0000 7f1fa3fff640 1 -- 172.21.15.100:0/4115970857 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f1fac1370e0 con 0x7f1fac138250 2024-09-13T11:55:05.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.849+0000 7f1fa3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:05.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:05.849+0000 7f1fa3fff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:55:06.090 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:06.088+0000 7f1fb13ec640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:55:06.090 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:06.088+0000 7f1fb13ec640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:55:06.090 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:06.088+0000 7f1fb13ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:06.090 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:06.089+0000 7f1fb13ec640 1 -- 172.21.15.100:0/4115970857 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1fac0a0950 con 0x7f1fac138250 2024-09-13T11:55:06.091 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:06.089+0000 7f1fa3fff640 1 -- 172.21.15.100:0/4115970857 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f1fa4051b40 con 0x7f1fac138250 2024-09-13T11:55:06.091 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:06.089+0000 7f1fa3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:55:06.091 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:06.089+0000 7f1fa3fff640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:55:06.091 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:55:06.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:06.091+0000 7f1fb13ec640 1 -- 172.21.15.100:0/4115970857 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f1f8c03df50 msgr2=0x7f1f8c040410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:06.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:06.091+0000 7f1fb13ec640 1 --2- 172.21.15.100:0/4115970857 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f1f8c03df50 0x7f1f8c040410 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c034000 tx=0x7f1f9c02e000 comp rx=0 tx=0).stop 2024-09-13T11:55:06.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:06.091+0000 7f1fb13ec640 10 monclient: shutdown 2024-09-13T11:55:06.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:06.091+0000 7f1fb13ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:55:06.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:06.091+0000 7f1fb13ec640 1 -- 172.21.15.100:0/4115970857 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fac138250 msgr2=0x7f1fac1205a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:06.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:06.091+0000 7f1fb13ec640 1 --2- 172.21.15.100:0/4115970857 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fac138250 0x7f1fac1205a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1fa400cef0 tx=0x7f1fa400a710 comp rx=0 tx=0).stop 2024-09-13T11:55:06.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:06.091+0000 7f1fb13ec640 1 -- 172.21.15.100:0/4115970857 shutdown_connections 2024-09-13T11:55:06.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:06.091+0000 7f1fb13ec640 1 --2- 172.21.15.100:0/4115970857 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f1f8c03df50 0x7f1f8c040410 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:06.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:06.091+0000 7f1fb13ec640 1 --2- 172.21.15.100:0/4115970857 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fac138250 0x7f1fac1205a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:06.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:06.091+0000 7f1fb13ec640 1 --2- 172.21.15.100:0/4115970857 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fac137e40 0x7f1fac136ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:06.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:06.091+0000 7f1fb13ec640 1 --2- 172.21.15.100:0/4115970857 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1fac137a60 0x7f1fac136590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:06.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:06.091+0000 7f1fb13ec640 1 -- 172.21.15.100:0/4115970857 >> 172.21.15.100:0/4115970857 conn(0x7f1fac0aa580 msgr2=0x7f1fac0a2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:55:06.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:06.091+0000 7f1fb13ec640 1 -- 172.21.15.100:0/4115970857 shutdown_connections 2024-09-13T11:55:06.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:06.091+0000 7f1fb13ec640 1 -- 172.21.15.100:0/4115970857 wait complete. 2024-09-13T11:55:09.104 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:55:09.213 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.210+0000 7f4be9b56640 10 monclient: get_monmap_and_config 2024-09-13T11:55:09.213 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.210+0000 7f4be9b56640 10 monclient: build_initial_monmap 2024-09-13T11:55:09.213 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.210+0000 7f4be9b56640 10 monclient: monmap: 2024-09-13T11:55:09.213 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:55:09.216 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:09.216 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:55:09.212595+0000 2024-09-13T11:55:09.216 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:55:09.212595+0000 2024-09-13T11:55:09.216 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:55:09.216 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:09.216 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:55:09.216 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:55:09.216 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:55:09.216 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:55:09.216 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:55:09.216 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:09.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.211+0000 7f4be9b56640 1 Processor -- start 2024-09-13T11:55:09.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.211+0000 7f4be9b56640 1 -- start start 2024-09-13T11:55:09.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.211+0000 7f4be9b56640 10 monclient: init 2024-09-13T11:55:09.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4be9b56640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:09.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4be9b56640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-13T11:55:09.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4be9b56640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4be40a2fe0 0x7f4be409f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:09.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4be9b56640 10 monclient(hunting): picked mon.noname-c con 0x7f4be40a2fe0 addr v2:172.21.15.122:3300/0 2024-09-13T11:55:09.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4be9b56640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f4be4133c70 0x7f4be409ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:09.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4be9b56640 10 monclient(hunting): picked mon.noname-a con 0x7f4be4133c70 addr v2:172.21.15.100:3300/0 2024-09-13T11:55:09.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4be9b56640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4be409bf40 addr v1:172.21.15.122:6789/0 2024-09-13T11:55:09.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4be9b56640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4be40a2030 con 0x7f4be409bf40 2024-09-13T11:55:09.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4be9b56640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:09.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4be9b56640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f4be409e190 con 0x7f4be4133c70 2024-09-13T11:55:09.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4be9b56640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:09.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4be9b56640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f4be40a33c0 con 0x7f4be40a2fe0 2024-09-13T11:55:09.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4be9b56640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:09.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4be9b56640 10 monclient(hunting): authenticate will time out at 889.806260s 2024-09-13T11:55:09.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4bdbfff640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f4be4133c70 0x7f4be409ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:09.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4bdbfff640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f4be4133c70 0x7f4be409ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47056/0 (socket says 172.21.15.100:47056) 2024-09-13T11:55:09.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4bdbfff640 1 -- 172.21.15.100:0/3215641512 learned_addr learned my addr 172.21.15.100:0/3215641512 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:55:09.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4bdbfff640 10 monclient(hunting): get_auth_request con 0x7f4be4133c70 auth_method 0 2024-09-13T11:55:09.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4bdbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:09.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4bdbfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:09.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4bdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:09.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4bdbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:09.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4bdbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:09.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4bdbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:09.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4be37fe640 1 --2- 172.21.15.100:0/3215641512 >> v2:172.21.15.122:3300/0 conn(0x7f4be40a2fe0 0x7f4be409f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:09.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4be37fe640 10 monclient(hunting): get_auth_request con 0x7f4be40a2fe0 auth_method 0 2024-09-13T11:55:09.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4be37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:09.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4be37fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:09.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4be37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:09.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4bdbfff640 10 monclient(hunting): handle_auth_done global_id 14214 payload 274 2024-09-13T11:55:09.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4bdbfff640 1 -- 172.21.15.100:0/3215641512 >> v2:172.21.15.122:3300/0 conn(0x7f4be40a2fe0 msgr2=0x7f4be409f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:55:09.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4bdbfff640 1 --2- 172.21.15.100:0/3215641512 >> v2:172.21.15.122:3300/0 conn(0x7f4be40a2fe0 0x7f4be409f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:09.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4bdbfff640 1 -- 172.21.15.100:0/3215641512 >> v1:172.21.15.122:6789/0 conn(0x7f4be409bf40 legacy=0x7f4be413ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:55:09.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4bdbfff640 10 monclient: _finish_hunting 0 2024-09-13T11:55:09.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4bdbfff640 1 monclient: found mon.noname-a 2024-09-13T11:55:09.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4bdbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:09.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4bdbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:55:09.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.212+0000 7f4bdbfff640 1 -- 172.21.15.100:0/3215641512 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be409dbd0 con 0x7f4be4133c70 2024-09-13T11:55:09.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.213+0000 7f4bdbfff640 1 --2- 172.21.15.100:0/3215641512 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4be4133c70 0x7f4be409ba00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4be4076dc0 tx=0x7f4bcc02c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5009bd5f9eb9f4cd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:09.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.213+0000 7f4be2ffd640 1 -- 172.21.15.100:0/3215641512 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc038070 con 0x7f4be4133c70 2024-09-13T11:55:09.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.213+0000 7f4be2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:09.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.213+0000 7f4be2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:55:09.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.213+0000 7f4be2ffd640 10 monclient: dump: 2024-09-13T11:55:09.228 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:09.229 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:09.229 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:09.229 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:09.229 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:09.229 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:09.229 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:09.229 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:09.229 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:09.229 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:09.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.213+0000 7f4be2ffd640 10 monclient: _finish_auth 0 2024-09-13T11:55:09.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.213+0000 7f4be2ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:55:09.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.213+0000 7f4be2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:09.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.213+0000 7f4be9b56640 5 monclient: authenticate success, global_id 14214 2024-09-13T11:55:09.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.213+0000 7f4be9b56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:55:09.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.213+0000 7f4be2ffd640 1 -- 172.21.15.100:0/3215641512 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bcc031710 con 0x7f4be4133c70 2024-09-13T11:55:09.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.213+0000 7f4be2ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:55:09.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.213+0000 7f4be9b56640 10 monclient: get_monmap_and_config success 2024-09-13T11:55:09.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.213+0000 7f4be9b56640 4 set_mon_vals no callback set 2024-09-13T11:55:09.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.213+0000 7f4be2ffd640 1 -- 172.21.15.100:0/3215641512 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc031ae0 con 0x7f4be4133c70 2024-09-13T11:55:09.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.213+0000 7f4be2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:09.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.213+0000 7f4be2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:09.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.213+0000 7f4be2ffd640 10 monclient: dump: 2024-09-13T11:55:09.230 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:09.230 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:09.230 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:09.230 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:09.230 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:09.230 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:09.231 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:09.231 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:09.231 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:09.231 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:09.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.214+0000 7f4be9b56640 10 monclient: shutdown 2024-09-13T11:55:09.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.214+0000 7f4be9b56640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:55:09.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.214+0000 7f4be9b56640 1 -- 172.21.15.100:0/3215641512 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4be4133c70 msgr2=0x7f4be409ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:09.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.214+0000 7f4be9b56640 1 --2- 172.21.15.100:0/3215641512 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4be4133c70 0x7f4be409ba00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4be4076dc0 tx=0x7f4bcc02c770 comp rx=0 tx=0).stop 2024-09-13T11:55:09.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.214+0000 7f4be9b56640 1 -- 172.21.15.100:0/3215641512 shutdown_connections 2024-09-13T11:55:09.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.214+0000 7f4be9b56640 1 --2- 172.21.15.100:0/3215641512 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4be4133c70 0x7f4be409ba00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:09.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.214+0000 7f4be9b56640 1 --2- 172.21.15.100:0/3215641512 >> v2:172.21.15.122:3300/0 conn(0x7f4be40a2fe0 0x7f4be409f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:09.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.214+0000 7f4be9b56640 1 -- 172.21.15.100:0/3215641512 >> 172.21.15.100:0/3215641512 conn(0x7f4be40a6470 msgr2=0x7f4be40a2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:55:09.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.214+0000 7f4be9b56640 1 -- 172.21.15.100:0/3215641512 shutdown_connections 2024-09-13T11:55:09.232 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.214+0000 7f4be9b56640 1 -- 172.21.15.100:0/3215641512 wait complete. 2024-09-13T11:55:09.232 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.214+0000 7f4be9b56640 10 monclient: build_initial_monmap 2024-09-13T11:55:09.232 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.214+0000 7f4be9b56640 10 monclient: monmap: 2024-09-13T11:55:09.232 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:55:09.232 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:55:09.232 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:55:09.232 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:55:09.232 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:55:09.232 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:09.232 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:55:09.232 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:55:09.232 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:55:09.232 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:09.232 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be9b56640 1 Processor -- start 2024-09-13T11:55:09.233 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be9b56640 1 -- start start 2024-09-13T11:55:09.233 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be9b56640 10 monclient: init 2024-09-13T11:55:09.233 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be9b56640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:09.233 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be9b56640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:55:09.233 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be9b56640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4be409bf40 0x7f4be411cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:09.233 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be9b56640 10 monclient(hunting): picked mon.noname-a con 0x7f4be409bf40 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:55:09.233 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be9b56640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4be40a2fe0 0x7f4be411d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:09.233 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be9b56640 10 monclient(hunting): picked mon.noname-c con 0x7f4be40a2fe0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:55:09.233 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be9b56640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4be411d5a0 0x7f4be41c86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:09.233 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be9b56640 10 monclient(hunting): picked mon.noname-b con 0x7f4be411d5a0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:55:09.233 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be9b56640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:09.234 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be9b56640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f4be4134590 con 0x7f4be409bf40 2024-09-13T11:55:09.234 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be9b56640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:09.234 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be9b56640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f4be409cb30 con 0x7f4be411d5a0 2024-09-13T11:55:09.234 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be9b56640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:09.234 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be9b56640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f4be4076a90 con 0x7f4be40a2fe0 2024-09-13T11:55:09.234 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be9b56640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:09.234 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be9b56640 10 monclient(hunting): authenticate will time out at 889.809737s 2024-09-13T11:55:09.234 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4bdbfff640 1 -- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4be40a2fe0 msgr2=0x7f4be411d060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:3301/0 2024-09-13T11:55:09.234 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4bdbfff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4be40a2fe0 0x7f4be411d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:55:09.234 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be37fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4be409bf40 0x7f4be411cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:09.234 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be37fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4be409bf40 0x7f4be411cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47066/0 (socket says 172.21.15.100:47066) 2024-09-13T11:55:09.234 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be37fe640 1 -- 172.21.15.100:0/964749626 learned_addr learned my addr 172.21.15.100:0/964749626 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:55:09.234 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be37fe640 10 monclient(hunting): get_auth_request con 0x7f4be409bf40 auth_method 0 2024-09-13T11:55:09.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:09.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be37fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:09.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:09.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:09.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:09.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be3fff640 1 --2- 172.21.15.100:0/964749626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4be411d5a0 0x7f4be41c86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:09.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:09.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be3fff640 10 monclient(hunting): get_auth_request con 0x7f4be411d5a0 auth_method 0 2024-09-13T11:55:09.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:09.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be3fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:09.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.215+0000 7f4be3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:09.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:09.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:09.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:09.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be37fe640 10 monclient(hunting): handle_auth_done global_id 14217 payload 824 2024-09-13T11:55:09.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be37fe640 1 -- 172.21.15.100:0/964749626 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4be40a2fe0 msgr2=0x7f4be411d060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:55:09.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be37fe640 1 --2- 172.21.15.100:0/964749626 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4be40a2fe0 0x7f4be411d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:09.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be37fe640 1 -- 172.21.15.100:0/964749626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4be411d5a0 msgr2=0x7f4be41c86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:09.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be37fe640 1 --2- 172.21.15.100:0/964749626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4be411d5a0 0x7f4be41c86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:09.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be37fe640 10 monclient: _finish_hunting 0 2024-09-13T11:55:09.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be37fe640 1 monclient: found mon.noname-a 2024-09-13T11:55:09.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:09.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:55:09.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be37fe640 1 -- 172.21.15.100:0/964749626 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be411fa80 con 0x7f4be409bf40 2024-09-13T11:55:09.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be37fe640 1 --2- 172.21.15.100:0/964749626 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4be409bf40 0x7f4be411cb20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4bd400d800 tx=0x7f4bd400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:09.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be0ff9640 1 -- 172.21.15.100:0/964749626 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd400b6c0 con 0x7f4be409bf40 2024-09-13T11:55:09.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:09.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:55:09.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be0ff9640 10 monclient: dump: 2024-09-13T11:55:09.237 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:09.237 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:09.237 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:09.237 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:09.237 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:09.237 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:09.237 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:09.237 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:09.237 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:09.237 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:09.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be0ff9640 10 monclient: _finish_auth 0 2024-09-13T11:55:09.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be0ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:55:09.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:09.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be0ff9640 1 -- 172.21.15.100:0/964749626 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bd4019070 con 0x7f4be409bf40 2024-09-13T11:55:09.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be9b56640 5 monclient: authenticate success, global_id 14217 2024-09-13T11:55:09.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be9b56640 10 monclient: _renew_subs 2024-09-13T11:55:09.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be9b56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:09.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be9b56640 1 -- 172.21.15.100:0/964749626 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4be411e590 con 0x7f4be409bf40 2024-09-13T11:55:09.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be0ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:55:09.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be0ff9640 1 -- 172.21.15.100:0/964749626 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd400ba80 con 0x7f4be409bf40 2024-09-13T11:55:09.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:09.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:09.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be0ff9640 10 monclient: dump: 2024-09-13T11:55:09.238 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:09.238 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:09.238 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:09.239 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:09.239 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:09.239 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:09.239 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:09.239 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:09.239 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:09.239 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:09.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be9b56640 10 monclient: _renew_subs 2024-09-13T11:55:09.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be9b56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:09.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be27fc640 4 set_mon_vals no callback set 2024-09-13T11:55:09.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.216+0000 7f4be9b56640 1 -- 172.21.15.100:0/964749626 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4be4130230 con 0x7f4be409bf40 2024-09-13T11:55:09.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.217+0000 7f4be9b56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:09.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.217+0000 7f4be9b56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:09.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.217+0000 7f4be9b56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:09.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.217+0000 7f4be9b56640 1 -- 172.21.15.100:0/964749626 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4be40a3e40 con 0x7f4be409bf40 2024-09-13T11:55:09.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.217+0000 7f4be0ff9640 1 -- 172.21.15.100:0/964749626 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f4bd400bc20 con 0x7f4be409bf40 2024-09-13T11:55:09.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.217+0000 7f4be0ff9640 1 --2- 172.21.15.100:0/964749626 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f4bc403df50 0x7f4bc4040410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:09.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.217+0000 7f4be0ff9640 1 -- 172.21.15.100:0/964749626 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4346+0+0 (secure 0 0 0) 0x7f4bd4052990 con 0x7f4be409bf40 2024-09-13T11:55:09.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.218+0000 7f4bdbfff640 1 --2- 172.21.15.100:0/964749626 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f4bc403df50 0x7f4bc4040410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:09.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.218+0000 7f4bdbfff640 10 monclient: get_auth_request con 0x7f4bc403df50 auth_method 0 2024-09-13T11:55:09.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.218+0000 7f4bdbfff640 1 --2- 172.21.15.100:0/964749626 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f4bc403df50 0x7f4bc4040410 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc035000 tx=0x7f4bcc030000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:09.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.222+0000 7f4be0ff9640 1 -- 172.21.15.100:0/964749626 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f4be40a3e40 con 0x7f4be409bf40 2024-09-13T11:55:09.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.222+0000 7f4be0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:09.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.222+0000 7f4be0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:55:09.466 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.463+0000 7f4be9b56640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:55:09.466 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.463+0000 7f4be9b56640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:55:09.466 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.463+0000 7f4be9b56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:09.466 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.463+0000 7f4be9b56640 1 -- 172.21.15.100:0/964749626 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4be40a0000 con 0x7f4be409bf40 2024-09-13T11:55:09.466 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.463+0000 7f4be0ff9640 1 -- 172.21.15.100:0/964749626 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f4bd4011d50 con 0x7f4be409bf40 2024-09-13T11:55:09.466 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.463+0000 7f4be0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:55:09.466 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.463+0000 7f4be0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:55:09.467 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:55:09.468 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.465+0000 7f4be9b56640 1 -- 172.21.15.100:0/964749626 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f4bc403df50 msgr2=0x7f4bc4040410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:09.468 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.465+0000 7f4be9b56640 1 --2- 172.21.15.100:0/964749626 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f4bc403df50 0x7f4bc4040410 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc035000 tx=0x7f4bcc030000 comp rx=0 tx=0).stop 2024-09-13T11:55:09.468 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.465+0000 7f4be9b56640 10 monclient: shutdown 2024-09-13T11:55:09.468 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.465+0000 7f4be9b56640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:55:09.468 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.465+0000 7f4be9b56640 1 -- 172.21.15.100:0/964749626 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4be409bf40 msgr2=0x7f4be411cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:09.468 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.465+0000 7f4be9b56640 1 --2- 172.21.15.100:0/964749626 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4be409bf40 0x7f4be411cb20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4bd400d800 tx=0x7f4bd400dcc0 comp rx=0 tx=0).stop 2024-09-13T11:55:09.468 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.465+0000 7f4be9b56640 1 -- 172.21.15.100:0/964749626 shutdown_connections 2024-09-13T11:55:09.468 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.465+0000 7f4be9b56640 1 --2- 172.21.15.100:0/964749626 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f4bc403df50 0x7f4bc4040410 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:09.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.465+0000 7f4be9b56640 1 --2- 172.21.15.100:0/964749626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4be411d5a0 0x7f4be41c86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:09.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.465+0000 7f4be9b56640 1 --2- 172.21.15.100:0/964749626 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4be40a2fe0 0x7f4be411d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:09.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.465+0000 7f4be9b56640 1 --2- 172.21.15.100:0/964749626 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4be409bf40 0x7f4be411cb20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:09.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.465+0000 7f4be9b56640 1 -- 172.21.15.100:0/964749626 >> 172.21.15.100:0/964749626 conn(0x7f4be40a6470 msgr2=0x7f4be40a0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:55:09.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.465+0000 7f4be9b56640 1 -- 172.21.15.100:0/964749626 shutdown_connections 2024-09-13T11:55:09.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:09.465+0000 7f4be9b56640 1 -- 172.21.15.100:0/964749626 wait complete. 2024-09-13T11:55:12.480 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:55:12.588 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.585+0000 7fd7cf6be640 10 monclient: get_monmap_and_config 2024-09-13T11:55:12.588 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.585+0000 7fd7cf6be640 10 monclient: build_initial_monmap 2024-09-13T11:55:12.589 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.586+0000 7fd7cf6be640 10 monclient: monmap: 2024-09-13T11:55:12.589 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:55:12.589 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:12.589 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:55:12.587859+0000 2024-09-13T11:55:12.589 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:55:12.587859+0000 2024-09-13T11:55:12.590 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:55:12.590 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:12.590 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:55:12.590 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:55:12.590 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:55:12.591 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:55:12.591 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:55:12.591 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:12.597 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.586+0000 7fd7cf6be640 1 Processor -- start 2024-09-13T11:55:12.597 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.586+0000 7fd7cf6be640 1 -- start start 2024-09-13T11:55:12.597 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.586+0000 7fd7cf6be640 10 monclient: init 2024-09-13T11:55:12.597 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cf6be640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:12.598 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cf6be640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-13T11:55:12.598 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cf6be640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd7c80a1d80 0x7fd7c809b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:12.598 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cf6be640 10 monclient(hunting): picked mon.noname-c con 0x7fd7c80a1d80 addr v2:172.21.15.122:3300/0 2024-09-13T11:55:12.598 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cf6be640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd7c80a2160 addr v1:172.21.15.122:6789/0 2024-09-13T11:55:12.599 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cf6be640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd7c813c9b0 0x7fd7c813eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:12.599 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cf6be640 10 monclient(hunting): picked mon.noname-a con 0x7fd7c813c9b0 addr v2:172.21.15.100:3300/0 2024-09-13T11:55:12.599 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cf6be640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7c809b640 con 0x7fd7c80a2160 2024-09-13T11:55:12.599 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cf6be640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:12.599 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cf6be640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7fd7c809a990 con 0x7fd7c813c9b0 2024-09-13T11:55:12.600 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cf6be640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:12.600 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cf6be640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7fd7c806d8c0 con 0x7fd7c80a1d80 2024-09-13T11:55:12.600 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cf6be640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:12.600 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cf6be640 10 monclient(hunting): authenticate will time out at 893.181320s 2024-09-13T11:55:12.600 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cdc5b640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd7c813c9b0 0x7fd7c813eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:12.600 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cdc5b640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd7c813c9b0 0x7fd7c813eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:43886/0 (socket says 172.21.15.100:43886) 2024-09-13T11:55:12.600 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cdc5b640 1 -- 172.21.15.100:0/2545093737 learned_addr learned my addr 172.21.15.100:0/2545093737 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:55:12.600 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cdc5b640 10 monclient(hunting): get_auth_request con 0x7fd7c813c9b0 auth_method 0 2024-09-13T11:55:12.600 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cdc5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:12.600 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cdc5b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:12.600 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cdc5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:12.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cdc5b640 1 -- 172.21.15.100:0/2545093737 >> v2:172.21.15.100:3300/0 conn(0x7fd7c813c9b0 msgr2=0x7fd7c813eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-13T11:55:12.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cdc5b640 1 -- 172.21.15.100:0/2545093737 >> v2:172.21.15.100:3300/0 conn(0x7fd7c813c9b0 msgr2=0x7fd7c813eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:55:12.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cdc5b640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.100:3300/0 conn(0x7fd7c813c9b0 0x7fd7c813eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:55:12.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cdc5b640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.100:3300/0 conn(0x7fd7c813c9b0 0x7fd7c813eda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:55:12.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cd45a640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.122:3300/0 conn(0x7fd7c80a1d80 0x7fd7c809b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:12.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cd45a640 10 monclient(hunting): get_auth_request con 0x7fd7c80a1d80 auth_method 0 2024-09-13T11:55:12.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cd45a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:12.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cd45a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:12.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cd45a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:12.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cd45a640 1 -- 172.21.15.100:0/2545093737 >> v2:172.21.15.122:3300/0 conn(0x7fd7c80a1d80 msgr2=0x7fd7c809b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:55:12.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cd45a640 1 -- 172.21.15.100:0/2545093737 >> v2:172.21.15.122:3300/0 conn(0x7fd7c80a1d80 msgr2=0x7fd7c809b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:55:12.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cd45a640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.122:3300/0 conn(0x7fd7c80a1d80 0x7fd7c809b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:55:12.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.587+0000 7fd7cd45a640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.122:3300/0 conn(0x7fd7c80a1d80 0x7fd7c809b100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:55:12.791 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.788+0000 7fd7cdc5b640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.100:3300/0 conn(0x7fd7c813c9b0 0x7fd7c813eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:12.791 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.788+0000 7fd7cdc5b640 10 monclient(hunting): get_auth_request con 0x7fd7c813c9b0 auth_method 0 2024-09-13T11:55:12.791 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.788+0000 7fd7cdc5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:12.791 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.788+0000 7fd7cdc5b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:12.791 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.788+0000 7fd7cdc5b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:55:12.791 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.788+0000 7fd7cdc5b640 1 -- 172.21.15.100:0/2545093737 >> v2:172.21.15.100:3300/0 conn(0x7fd7c813c9b0 msgr2=0x7fd7c813eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:55:12.791 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.788+0000 7fd7cdc5b640 1 -- 172.21.15.100:0/2545093737 >> v2:172.21.15.100:3300/0 conn(0x7fd7c813c9b0 msgr2=0x7fd7c813eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:55:12.791 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.788+0000 7fd7cdc5b640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.100:3300/0 conn(0x7fd7c813c9b0 0x7fd7c813eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:55:12.792 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.788+0000 7fd7cdc5b640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.100:3300/0 conn(0x7fd7c813c9b0 0x7fd7c813eda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T11:55:12.792 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.788+0000 7fd7cd45a640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.122:3300/0 conn(0x7fd7c80a1d80 0x7fd7c809b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:12.792 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.788+0000 7fd7cd45a640 10 monclient(hunting): get_auth_request con 0x7fd7c80a1d80 auth_method 0 2024-09-13T11:55:12.792 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.788+0000 7fd7cd45a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:12.792 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.788+0000 7fd7cd45a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:12.792 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.788+0000 7fd7cd45a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:55:12.792 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.788+0000 7fd7cd45a640 1 -- 172.21.15.100:0/2545093737 >> v2:172.21.15.122:3300/0 conn(0x7fd7c80a1d80 msgr2=0x7fd7c809b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-13T11:55:12.792 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.788+0000 7fd7cd45a640 1 -- 172.21.15.100:0/2545093737 >> v2:172.21.15.122:3300/0 conn(0x7fd7c80a1d80 msgr2=0x7fd7c809b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:55:12.792 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.788+0000 7fd7cd45a640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.122:3300/0 conn(0x7fd7c80a1d80 0x7fd7c809b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:55:12.792 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:12.788+0000 7fd7cd45a640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.122:3300/0 conn(0x7fd7c80a1d80 0x7fd7c809b100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T11:55:13.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.189+0000 7fd7cdc5b640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.100:3300/0 conn(0x7fd7c813c9b0 0x7fd7c813eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:13.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.189+0000 7fd7cdc5b640 10 monclient(hunting): get_auth_request con 0x7fd7c813c9b0 auth_method 0 2024-09-13T11:55:13.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.189+0000 7fd7cdc5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:13.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.189+0000 7fd7cdc5b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:13.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.189+0000 7fd7cdc5b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:55:13.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.189+0000 7fd7cdc5b640 1 -- 172.21.15.100:0/2545093737 >> v2:172.21.15.100:3300/0 conn(0x7fd7c813c9b0 msgr2=0x7fd7c813eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:55:13.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.189+0000 7fd7cdc5b640 1 -- 172.21.15.100:0/2545093737 >> v2:172.21.15.100:3300/0 conn(0x7fd7c813c9b0 msgr2=0x7fd7c813eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:55:13.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.189+0000 7fd7cdc5b640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.100:3300/0 conn(0x7fd7c813c9b0 0x7fd7c813eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:55:13.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.189+0000 7fd7cdc5b640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.100:3300/0 conn(0x7fd7c813c9b0 0x7fd7c813eda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T11:55:13.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.189+0000 7fd7cd45a640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.122:3300/0 conn(0x7fd7c80a1d80 0x7fd7c809b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:13.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.189+0000 7fd7cd45a640 10 monclient(hunting): get_auth_request con 0x7fd7c80a1d80 auth_method 0 2024-09-13T11:55:13.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.189+0000 7fd7cd45a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:13.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.189+0000 7fd7cd45a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:13.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.189+0000 7fd7cd45a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:55:13.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.189+0000 7fd7cd45a640 1 -- 172.21.15.100:0/2545093737 >> v2:172.21.15.122:3300/0 conn(0x7fd7c80a1d80 msgr2=0x7fd7c809b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:55:13.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.189+0000 7fd7cd45a640 1 -- 172.21.15.100:0/2545093737 >> v2:172.21.15.122:3300/0 conn(0x7fd7c80a1d80 msgr2=0x7fd7c809b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:55:13.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.189+0000 7fd7cd45a640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.122:3300/0 conn(0x7fd7c80a1d80 0x7fd7c809b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:55:13.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.189+0000 7fd7cd45a640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.122:3300/0 conn(0x7fd7c80a1d80 0x7fd7c809b100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T11:55:13.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.991+0000 7fd7cdc5b640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.100:3300/0 conn(0x7fd7c813c9b0 0x7fd7c813eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:13.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.991+0000 7fd7cdc5b640 10 monclient(hunting): get_auth_request con 0x7fd7c813c9b0 auth_method 0 2024-09-13T11:55:13.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.991+0000 7fd7cdc5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:13.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.991+0000 7fd7cdc5b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:13.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.991+0000 7fd7cdc5b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:55:13.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.991+0000 7fd7cdc5b640 1 -- 172.21.15.100:0/2545093737 >> v2:172.21.15.100:3300/0 conn(0x7fd7c813c9b0 msgr2=0x7fd7c813eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:55:13.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.991+0000 7fd7cdc5b640 1 -- 172.21.15.100:0/2545093737 >> v2:172.21.15.100:3300/0 conn(0x7fd7c813c9b0 msgr2=0x7fd7c813eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:55:13.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.991+0000 7fd7cdc5b640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.100:3300/0 conn(0x7fd7c813c9b0 0x7fd7c813eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:55:13.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.991+0000 7fd7cdc5b640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.100:3300/0 conn(0x7fd7c813c9b0 0x7fd7c813eda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T11:55:13.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.991+0000 7fd7cd45a640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.122:3300/0 conn(0x7fd7c80a1d80 0x7fd7c809b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:13.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.991+0000 7fd7cd45a640 10 monclient(hunting): get_auth_request con 0x7fd7c80a1d80 auth_method 0 2024-09-13T11:55:13.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.991+0000 7fd7cd45a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:13.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.991+0000 7fd7cd45a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:13.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.991+0000 7fd7cd45a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:55:13.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.991+0000 7fd7cd45a640 1 -- 172.21.15.100:0/2545093737 >> v2:172.21.15.122:3300/0 conn(0x7fd7c80a1d80 msgr2=0x7fd7c809b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:55:13.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.991+0000 7fd7cd45a640 1 -- 172.21.15.100:0/2545093737 >> v2:172.21.15.122:3300/0 conn(0x7fd7c80a1d80 msgr2=0x7fd7c809b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:55:13.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.991+0000 7fd7cd45a640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.122:3300/0 conn(0x7fd7c80a1d80 0x7fd7c809b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:55:13.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:13.991+0000 7fd7cd45a640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.122:3300/0 conn(0x7fd7c80a1d80 0x7fd7c809b100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T11:55:15.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:15.588+0000 7fd7baffd640 10 monclient(hunting): tick 2024-09-13T11:55:15.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:15.588+0000 7fd7baffd640 10 monclient(hunting): _check_auth_tickets 2024-09-13T11:55:15.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:15.588+0000 7fd7baffd640 1 monclient(hunting): continuing hunt 2024-09-13T11:55:15.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:15.588+0000 7fd7baffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-13T11:55:15.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:15.588+0000 7fd7baffd640 1 -- 172.21.15.100:0/2545093737 >> v2:172.21.15.122:3300/0 conn(0x7fd7c80a1d80 msgr2=0x7fd7c809b100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:55:15.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:15.588+0000 7fd7baffd640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.122:3300/0 conn(0x7fd7c80a1d80 0x7fd7c809b100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:15.591 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:15.588+0000 7fd7baffd640 1 -- 172.21.15.100:0/2545093737 >> v2:172.21.15.100:3300/0 conn(0x7fd7c813c9b0 msgr2=0x7fd7c813eda0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:55:15.591 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:15.588+0000 7fd7baffd640 1 --2- 172.21.15.100:0/2545093737 >> v2:172.21.15.100:3300/0 conn(0x7fd7c813c9b0 0x7fd7c813eda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:15.591 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:15.588+0000 7fd7baffd640 1 -- 172.21.15.100:0/2545093737 >> v1:172.21.15.122:6789/0 conn(0x7fd7c80a2160 legacy=0x7fd7c80a2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:15.591 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:15.588+0000 7fd7baffd640 10 monclient: _add_conns ranks=[2,0] 2024-09-13T11:55:15.591 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:15.588+0000 7fd7baffd640 1 --2- 172.21.15.100:0/2545093737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7ac0036b0 0x7fd7ac005ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:15.591 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:15.588+0000 7fd7baffd640 10 monclient(hunting): picked mon.noname-b con 0x7fd7ac0036b0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:55:15.591 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:15.588+0000 7fd7baffd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd7ac006210 addr v1:172.21.15.100:6789/0 2024-09-13T11:55:15.591 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:15.588+0000 7fd7baffd640 1 -- 172.21.15.100:0/2545093737 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7c809a6b0 con 0x7fd7ac006210 2024-09-13T11:55:15.591 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:15.588+0000 7fd7baffd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:15.591 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:15.588+0000 7fd7baffd640 1 -- 172.21.15.100:0/2545093737 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fd7c809a990 con 0x7fd7ac0036b0 2024-09-13T11:55:15.592 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:15.588+0000 7fd7baffd640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:15.592 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:15.588+0000 7fd7cd45a640 1 -- 172.21.15.100:0/2545093737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7ac0036b0 msgr2=0x7fd7ac005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.100:3301/0 2024-09-13T11:55:15.592 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:15.588+0000 7fd7cd45a640 1 --2- 172.21.15.100:0/2545093737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7ac0036b0 0x7fd7ac005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:55:15.791 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:15.790+0000 7fd7cd45a640 1 -- 172.21.15.100:0/2545093737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7ac0036b0 msgr2=0x7fd7ac005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.100:3301/0 2024-09-13T11:55:15.791 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:15.790+0000 7fd7cd45a640 1 --2- 172.21.15.100:0/2545093737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7ac0036b0 0x7fd7ac005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T11:55:16.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.191+0000 7fd7cd45a640 1 -- 172.21.15.100:0/2545093737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7ac0036b0 msgr2=0x7fd7ac005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.100:3301/0 2024-09-13T11:55:16.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.191+0000 7fd7cd45a640 1 --2- 172.21.15.100:0/2545093737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7ac0036b0 0x7fd7ac005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T11:55:16.347 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.346+0000 7fd7bbfff640 1 -- 172.21.15.100:0/2545093737 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1081644301 0 0) 0x7fd7c809a6b0 con 0x7fd7ac006210 2024-09-13T11:55:16.348 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.346+0000 7fd7bbfff640 10 monclient(hunting): _negotiate 2024-09-13T11:55:16.348 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.346+0000 7fd7bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:16.348 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.346+0000 7fd7bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:16.348 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.346+0000 7fd7bbfff640 10 monclient(hunting): my global_id is 14220 2024-09-13T11:55:16.348 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.346+0000 7fd7bbfff640 1 -- 172.21.15.100:0/2545093737 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7b4003520 con 0x7fd7ac006210 2024-09-13T11:55:16.354 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 1 -- 172.21.15.100:0/2545093737 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3954842626 0 0) 0x7fd7b4003520 con 0x7fd7ac006210 2024-09-13T11:55:16.354 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 1 -- 172.21.15.100:0/2545093737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7ac0036b0 msgr2=0x7fd7ac005ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:55:16.355 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 1 --2- 172.21.15.100:0/2545093737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7ac0036b0 0x7fd7ac005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.355 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 10 monclient: _finish_hunting 0 2024-09-13T11:55:16.355 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 1 monclient: found mon.noname-a-legacy 2024-09-13T11:55:16.355 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:16.355 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.100:6789/0 2024-09-13T11:55:16.355 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 1 -- 172.21.15.100:0/2545093737 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ac00b0c0 con 0x7fd7ac006210 2024-09-13T11:55:16.355 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 10 monclient: _finish_auth 0 2024-09-13T11:55:16.355 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 10 monclient: _check_auth_tickets 2024-09-13T11:55:16.355 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:16.355 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 1 -- 172.21.15.100:0/2545093737 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 626700363 0 0) 0x7fd7c4001770 con 0x7fd7ac006210 2024-09-13T11:55:16.355 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:16.355 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-13T11:55:16.356 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 10 monclient: dump: 2024-09-13T11:55:16.356 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:16.356 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:16.356 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.356 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.356 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:16.356 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:16.356 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:16.356 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:16.356 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:16.356 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:16.356 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] but i'm connected to v1:172.21.15.100:6789/0, reconnecting 2024-09-13T11:55:16.356 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:16.357 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 1 -- 172.21.15.100:0/2545093737 >> v1:172.21.15.100:6789/0 conn(0x7fd7ac006210 legacy=0x7fd7ac0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:16.357 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:55:16.357 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7cdc5b640 1 -- 172.21.15.100:0/2545093737 reap_dead start 2024-09-13T11:55:16.357 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 1 --2- 172.21.15.100:0/2545093737 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd7b40039b0 0x7fd7b4003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:16.357 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 10 monclient(hunting): picked mon.a con 0x7fd7b40039b0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:55:16.357 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 1 --2- 172.21.15.100:0/2545093737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7b4004800 0x7fd7b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:16.357 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 10 monclient(hunting): picked mon.b con 0x7fd7b4004800 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:55:16.357 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 1 --2- 172.21.15.100:0/2545093737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7b4005370 0x7fd7b400b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:16.357 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 10 monclient(hunting): picked mon.c con 0x7fd7b4005370 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:55:16.357 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:16.357 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 1 -- 172.21.15.100:0/2545093737 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c809a990 con 0x7fd7b40039b0 2024-09-13T11:55:16.357 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:16.357 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 1 -- 172.21.15.100:0/2545093737 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b400c2c0 con 0x7fd7b4004800 2024-09-13T11:55:16.358 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:16.358 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 1 -- 172.21.15.100:0/2545093737 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fd7b400c4c0 con 0x7fd7b4005370 2024-09-13T11:55:16.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:16.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7bbfff640 10 monclient(hunting): _finish_auth 0 2024-09-13T11:55:16.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7cdc5b640 1 -- 172.21.15.100:0/2545093737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7b4005370 msgr2=0x7fd7b400b9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.100:3301/0 2024-09-13T11:55:16.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7cd45a640 1 --2- 172.21.15.100:0/2545093737 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd7b40039b0 0x7fd7b4003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:16.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7cdc5b640 1 --2- 172.21.15.100:0/2545093737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7b4005370 0x7fd7b400b9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:55:16.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7cd45a640 10 monclient(hunting): get_auth_request con 0x7fd7b40039b0 auth_method 0 2024-09-13T11:55:16.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7cd45a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:16.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7cd45a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:16.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.351+0000 7fd7cd45a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:55:16.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7ccc59640 1 --2- 172.21.15.100:0/2545093737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7b4004800 0x7fd7b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:16.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7cd45a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:16.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7cd45a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:16.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7cd45a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:55:16.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7ccc59640 10 monclient(hunting): get_auth_request con 0x7fd7b4004800 auth_method 0 2024-09-13T11:55:16.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7ccc59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:16.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7ccc59640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:16.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7ccc59640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:55:16.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7ccc59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:16.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7ccc59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:16.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7ccc59640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:55:16.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7cd45a640 10 monclient(hunting): handle_auth_done global_id 14220 payload 293 2024-09-13T11:55:16.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7cd45a640 1 -- 172.21.15.100:0/2545093737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7b4005370 msgr2=0x7fd7b400b9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:55:16.361 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7cd45a640 1 --2- 172.21.15.100:0/2545093737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7b4005370 0x7fd7b400b9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.361 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7cd45a640 1 -- 172.21.15.100:0/2545093737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7b4004800 msgr2=0x7fd7b4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:55:16.361 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7cd45a640 1 --2- 172.21.15.100:0/2545093737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7b4004800 0x7fd7b4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.361 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7cd45a640 10 monclient: _finish_hunting 0 2024-09-13T11:55:16.361 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7cd45a640 1 monclient: found mon.a 2024-09-13T11:55:16.361 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7cd45a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:16.361 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7cd45a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:16.361 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7cd45a640 1 -- 172.21.15.100:0/2545093737 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b4003520 con 0x7fd7b40039b0 2024-09-13T11:55:16.361 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7cd45a640 10 monclient: _finish_auth 0 2024-09-13T11:55:16.361 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7cd45a640 10 monclient: _check_auth_tickets 2024-09-13T11:55:16.362 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7cd45a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:16.362 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7cf6be640 5 monclient: authenticate success, global_id 14220 2024-09-13T11:55:16.362 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.352+0000 7fd7cf6be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:55:16.362 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.353+0000 7fd7cd45a640 1 --2- 172.21.15.100:0/2545093737 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd7b40039b0 0x7fd7b4003db0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc003a10 tx=0x7fd7b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10184cd316568027 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:16.370 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.355+0000 7fd7bbfff640 1 -- 172.21.15.100:0/2545093737 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc031070 con 0x7fd7b40039b0 2024-09-13T11:55:16.370 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.355+0000 7fd7bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:16.370 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.355+0000 7fd7bbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:16.370 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.355+0000 7fd7bbfff640 10 monclient: dump: 2024-09-13T11:55:16.370 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:16.370 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:16.370 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.370 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.370 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:16.370 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:16.370 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:16.370 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:16.370 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:16.371 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:16.371 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.355+0000 7fd7cf6be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:55:16.371 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.355+0000 7fd7bbfff640 1 -- 172.21.15.100:0/2545093737 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7bc02b990 con 0x7fd7b40039b0 2024-09-13T11:55:16.371 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.355+0000 7fd7bbfff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:55:16.371 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.355+0000 7fd7cf6be640 10 monclient: get_monmap_and_config success 2024-09-13T11:55:16.371 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.355+0000 7fd7bbfff640 1 -- 172.21.15.100:0/2545093737 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc003610 con 0x7fd7b40039b0 2024-09-13T11:55:16.371 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.355+0000 7fd7cf6be640 4 set_mon_vals no callback set 2024-09-13T11:55:16.371 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.355+0000 7fd7bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:16.371 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.355+0000 7fd7bbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:16.371 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.355+0000 7fd7bbfff640 10 monclient: dump: 2024-09-13T11:55:16.371 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:16.372 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:16.372 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.372 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.372 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:16.372 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:16.372 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:16.372 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:16.372 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:16.372 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:16.372 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.356+0000 7fd7cf6be640 10 monclient: shutdown 2024-09-13T11:55:16.372 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.356+0000 7fd7cf6be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:55:16.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.356+0000 7fd7cf6be640 1 -- 172.21.15.100:0/2545093737 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd7b40039b0 msgr2=0x7fd7b4003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:16.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.356+0000 7fd7cf6be640 1 --2- 172.21.15.100:0/2545093737 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd7b40039b0 0x7fd7b4003db0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc003a10 tx=0x7fd7b4005830 comp rx=0 tx=0).stop 2024-09-13T11:55:16.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.356+0000 7fd7cf6be640 1 -- 172.21.15.100:0/2545093737 shutdown_connections 2024-09-13T11:55:16.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.356+0000 7fd7cf6be640 1 --2- 172.21.15.100:0/2545093737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7b4005370 0x7fd7b400b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.356+0000 7fd7cf6be640 1 --2- 172.21.15.100:0/2545093737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7b4004800 0x7fd7b4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.356+0000 7fd7cf6be640 1 --2- 172.21.15.100:0/2545093737 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd7b40039b0 0x7fd7b4003db0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.356+0000 7fd7cf6be640 1 -- 172.21.15.100:0/2545093737 >> 172.21.15.100:0/2545093737 conn(0x7fd7c80a2e70 msgr2=0x7fd7c80a3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:55:16.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.356+0000 7fd7cf6be640 1 -- 172.21.15.100:0/2545093737 shutdown_connections 2024-09-13T11:55:16.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.356+0000 7fd7cf6be640 1 -- 172.21.15.100:0/2545093737 wait complete. 2024-09-13T11:55:16.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.357+0000 7fd7cf6be640 10 monclient: build_initial_monmap 2024-09-13T11:55:16.374 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.357+0000 7fd7cf6be640 10 monclient: monmap: 2024-09-13T11:55:16.374 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:55:16.374 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:55:16.374 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:55:16.374 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:55:16.374 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:55:16.374 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:16.374 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:55:16.374 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:55:16.374 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:55:16.374 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:16.374 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.357+0000 7fd7cf6be640 1 Processor -- start 2024-09-13T11:55:16.374 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.357+0000 7fd7cf6be640 1 -- start start 2024-09-13T11:55:16.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.357+0000 7fd7cf6be640 10 monclient: init 2024-09-13T11:55:16.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cf6be640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:16.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cf6be640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:55:16.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cf6be640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7b40039b0 0x7fd7c8139ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:16.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cf6be640 10 monclient(hunting): picked mon.noname-c con 0x7fd7b40039b0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:55:16.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cf6be640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd7b4004800 0x7fd7c813a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:16.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cf6be640 10 monclient(hunting): picked mon.noname-a con 0x7fd7b4004800 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:55:16.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cf6be640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7b4005370 0x7fd7c813a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:16.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cf6be640 10 monclient(hunting): picked mon.noname-b con 0x7fd7b4005370 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:55:16.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cf6be640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:16.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cf6be640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c80745d0 con 0x7fd7b4004800 2024-09-13T11:55:16.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cf6be640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:16.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cf6be640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c8074450 con 0x7fd7b4005370 2024-09-13T11:55:16.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cf6be640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:16.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cf6be640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fd7c809b100 con 0x7fd7b40039b0 2024-09-13T11:55:16.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cf6be640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:16.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cf6be640 10 monclient(hunting): authenticate will time out at 896.951217s 2024-09-13T11:55:16.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cd45a640 1 -- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7b40039b0 msgr2=0x7fd7c8139ec0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:3301/0 2024-09-13T11:55:16.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cd45a640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7b40039b0 0x7fd7c8139ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:55:16.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7ccc59640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd7b4004800 0x7fd7c813a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:16.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7ccc59640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd7b4004800 0x7fd7c813a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:43930/0 (socket says 172.21.15.100:43930) 2024-09-13T11:55:16.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7ccc59640 1 -- 172.21.15.100:0/1254069568 learned_addr learned my addr 172.21.15.100:0/1254069568 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:55:16.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7ccc59640 10 monclient(hunting): get_auth_request con 0x7fd7b4004800 auth_method 0 2024-09-13T11:55:16.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7ccc59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:16.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7ccc59640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:16.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7ccc59640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:16.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7ccc59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:16.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7ccc59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:16.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7ccc59640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:16.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cdc5b640 1 --2- 172.21.15.100:0/1254069568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7b4005370 0x7fd7c813a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:16.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cdc5b640 10 monclient(hunting): get_auth_request con 0x7fd7b4005370 auth_method 0 2024-09-13T11:55:16.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cdc5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:16.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cdc5b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:16.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cdc5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:16.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cdc5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:16.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cdc5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:16.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7cdc5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:16.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7ccc59640 10 monclient(hunting): handle_auth_done global_id 14223 payload 824 2024-09-13T11:55:16.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7ccc59640 1 -- 172.21.15.100:0/1254069568 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7b40039b0 msgr2=0x7fd7c8139ec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:55:16.378 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7ccc59640 1 --2- 172.21.15.100:0/1254069568 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7b40039b0 0x7fd7c8139ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.378 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7ccc59640 1 -- 172.21.15.100:0/1254069568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7b4005370 msgr2=0x7fd7c813a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:16.378 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7ccc59640 1 --2- 172.21.15.100:0/1254069568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7b4005370 0x7fd7c813a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7ccc59640 10 monclient: _finish_hunting 0 2024-09-13T11:55:16.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7ccc59640 1 monclient: found mon.noname-a 2024-09-13T11:55:16.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7ccc59640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:16.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7ccc59640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:55:16.381 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7ccc59640 1 -- 172.21.15.100:0/1254069568 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7c812d3a0 con 0x7fd7b4004800 2024-09-13T11:55:16.381 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.358+0000 7fd7ccc59640 1 --2- 172.21.15.100:0/1254069568 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd7b4004800 0x7fd7c813a400 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd7c400f8b0 tx=0x7fd7c400fd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:16.381 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.359+0000 7fd7b9ffb640 1 -- 172.21.15.100:0/1254069568 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c40052e0 con 0x7fd7b4004800 2024-09-13T11:55:16.381 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.359+0000 7fd7b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:16.381 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.359+0000 7fd7b9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:55:16.381 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.359+0000 7fd7b9ffb640 10 monclient: dump: 2024-09-13T11:55:16.381 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:16.381 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:16.381 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.381 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.381 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:16.381 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:16.382 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:16.382 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:16.382 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:16.382 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:16.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.359+0000 7fd7b9ffb640 10 monclient: _finish_auth 0 2024-09-13T11:55:16.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.359+0000 7fd7b9ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:55:16.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.359+0000 7fd7b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:16.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.359+0000 7fd7b9ffb640 1 -- 172.21.15.100:0/1254069568 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7c4018070 con 0x7fd7b4004800 2024-09-13T11:55:16.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.359+0000 7fd7cf6be640 5 monclient: authenticate success, global_id 14223 2024-09-13T11:55:16.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.359+0000 7fd7b9ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:55:16.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.359+0000 7fd7b9ffb640 1 -- 172.21.15.100:0/1254069568 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c400d970 con 0x7fd7b4004800 2024-09-13T11:55:16.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.359+0000 7fd7cf6be640 10 monclient: _renew_subs 2024-09-13T11:55:16.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.359+0000 7fd7cf6be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:16.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.359+0000 7fd7cf6be640 1 -- 172.21.15.100:0/1254069568 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7c80a4330 con 0x7fd7b4004800 2024-09-13T11:55:16.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.359+0000 7fd7b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:16.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.359+0000 7fd7bb7fe640 4 set_mon_vals no callback set 2024-09-13T11:55:16.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.359+0000 7fd7b9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:16.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.359+0000 7fd7b9ffb640 10 monclient: dump: 2024-09-13T11:55:16.383 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:16.383 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:16.383 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.383 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.383 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:16.383 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:16.383 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:16.384 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:16.384 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:16.384 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:16.384 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.359+0000 7fd7cf6be640 10 monclient: _renew_subs 2024-09-13T11:55:16.384 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.359+0000 7fd7cf6be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:16.384 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.359+0000 7fd7cf6be640 1 -- 172.21.15.100:0/1254069568 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7c812d0c0 con 0x7fd7b4004800 2024-09-13T11:55:16.384 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.360+0000 7fd7b9ffb640 1 -- 172.21.15.100:0/1254069568 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fd7c400db10 con 0x7fd7b4004800 2024-09-13T11:55:16.384 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.360+0000 7fd7b9ffb640 1 --2- 172.21.15.100:0/1254069568 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fd7ac03e810 0x7fd7ac040cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:16.384 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.360+0000 7fd7cf6be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:16.384 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.360+0000 7fd7cf6be640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:16.384 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.360+0000 7fd7cf6be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:16.384 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.360+0000 7fd7b9ffb640 1 -- 172.21.15.100:0/1254069568 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4346+0+0 (secure 0 0 0) 0x7fd7c4052560 con 0x7fd7b4004800 2024-09-13T11:55:16.384 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.360+0000 7fd7cf6be640 1 -- 172.21.15.100:0/1254069568 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7c80a2570 con 0x7fd7b4004800 2024-09-13T11:55:16.385 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.360+0000 7fd7cd45a640 1 --2- 172.21.15.100:0/1254069568 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fd7ac03e810 0x7fd7ac040cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:16.385 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.360+0000 7fd7cd45a640 10 monclient: get_auth_request con 0x7fd7ac03e810 auth_method 0 2024-09-13T11:55:16.385 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.361+0000 7fd7cd45a640 1 --2- 172.21.15.100:0/1254069568 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fd7ac03e810 0x7fd7ac040cd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc036000 tx=0x7fd7bc031000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:16.385 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.365+0000 7fd7b9ffb640 1 -- 172.21.15.100:0/1254069568 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fd7c809b640 con 0x7fd7b4004800 2024-09-13T11:55:16.385 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.365+0000 7fd7b9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:16.385 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.365+0000 7fd7b9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:55:16.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.608+0000 7fd7cf6be640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:55:16.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.608+0000 7fd7cf6be640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:55:16.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.608+0000 7fd7cf6be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:16.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.608+0000 7fd7cf6be640 1 -- 172.21.15.100:0/1254069568 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd7c812b720 con 0x7fd7b4004800 2024-09-13T11:55:16.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.608+0000 7fd7b9ffb640 1 -- 172.21.15.100:0/1254069568 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fd7c4003b10 con 0x7fd7b4004800 2024-09-13T11:55:16.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.608+0000 7fd7b9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:55:16.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.608+0000 7fd7b9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:55:16.612 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":20,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:55:16.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.610+0000 7fd7cf6be640 1 -- 172.21.15.100:0/1254069568 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fd7ac03e810 msgr2=0x7fd7ac040cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:16.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.610+0000 7fd7cf6be640 1 --2- 172.21.15.100:0/1254069568 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fd7ac03e810 0x7fd7ac040cd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc036000 tx=0x7fd7bc031000 comp rx=0 tx=0).stop 2024-09-13T11:55:16.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.610+0000 7fd7cf6be640 10 monclient: shutdown 2024-09-13T11:55:16.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.610+0000 7fd7cf6be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:55:16.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.610+0000 7fd7cf6be640 1 -- 172.21.15.100:0/1254069568 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd7b4004800 msgr2=0x7fd7c813a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:16.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.610+0000 7fd7cf6be640 1 --2- 172.21.15.100:0/1254069568 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd7b4004800 0x7fd7c813a400 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd7c400f8b0 tx=0x7fd7c400fd70 comp rx=0 tx=0).stop 2024-09-13T11:55:16.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.610+0000 7fd7cf6be640 1 -- 172.21.15.100:0/1254069568 shutdown_connections 2024-09-13T11:55:16.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.610+0000 7fd7cf6be640 1 --2- 172.21.15.100:0/1254069568 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fd7ac03e810 0x7fd7ac040cd0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.610+0000 7fd7cf6be640 1 --2- 172.21.15.100:0/1254069568 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd7b4004800 0x7fd7c813a400 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.610+0000 7fd7cf6be640 1 --2- 172.21.15.100:0/1254069568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7b4005370 0x7fd7c813a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.610+0000 7fd7cf6be640 1 --2- 172.21.15.100:0/1254069568 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7b40039b0 0x7fd7c8139ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.610+0000 7fd7cf6be640 1 -- 172.21.15.100:0/1254069568 >> 172.21.15.100:0/1254069568 conn(0x7fd7c80a2e70 msgr2=0x7fd7c8137c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:55:16.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.610+0000 7fd7cf6be640 1 -- 172.21.15.100:0/1254069568 shutdown_connections 2024-09-13T11:55:16.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.610+0000 7fd7cf6be640 1 -- 172.21.15.100:0/1254069568 wait complete. 2024-09-13T11:55:16.623 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-13T11:55:16.624 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-13T11:55:16.624 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-13T11:55:16.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.722+0000 7f782a9bb640 10 monclient: get_monmap_and_config 2024-09-13T11:55:16.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.722+0000 7f782a9bb640 10 monclient: build_initial_monmap 2024-09-13T11:55:16.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.722+0000 7f782a9bb640 10 monclient: monmap: 2024-09-13T11:55:16.723 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:55:16.724 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:16.724 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:55:16.723087+0000 2024-09-13T11:55:16.724 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:55:16.723087+0000 2024-09-13T11:55:16.724 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:55:16.724 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:16.724 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:55:16.724 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:55:16.724 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:55:16.724 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:55:16.724 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:55:16.724 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:16.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.722+0000 7f782a9bb640 1 Processor -- start 2024-09-13T11:55:16.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f782a9bb640 1 -- start start 2024-09-13T11:55:16.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f782a9bb640 10 monclient: init 2024-09-13T11:55:16.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f782a9bb640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:16.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f782a9bb640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-13T11:55:16.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f782a9bb640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78240a37d0 0x7f78240a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:16.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f782a9bb640 10 monclient(hunting): picked mon.noname-b con 0x7f78240a37d0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:55:16.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f782a9bb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f782406eeb0 0x7f78240a40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:16.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f782a9bb640 10 monclient(hunting): picked mon.noname-c con 0x7f782406eeb0 addr v2:172.21.15.122:3300/0 2024-09-13T11:55:16.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f782a9bb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7824137970 addr v1:172.21.15.122:6789/0 2024-09-13T11:55:16.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f782a9bb640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f782406e950 con 0x7f7824137970 2024-09-13T11:55:16.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f782a9bb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:16.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f782a9bb640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f782409fde0 con 0x7f782406eeb0 2024-09-13T11:55:16.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f782a9bb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:16.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f782a9bb640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f782406f330 con 0x7f78240a37d0 2024-09-13T11:55:16.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f782a9bb640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:16.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f782a9bb640 10 monclient(hunting): authenticate will time out at 897.316536s 2024-09-13T11:55:16.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f7823fff640 1 -- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78240a37d0 msgr2=0x7f78240a3bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.100:3301/0 2024-09-13T11:55:16.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f7823fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78240a37d0 0x7f78240a3bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:55:16.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f78237fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f782406eeb0 0x7f78240a40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:16.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f78237fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f782406eeb0 0x7f78240a40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.100:39580/0 (socket says 172.21.15.100:39580) 2024-09-13T11:55:16.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f78237fe640 1 -- 172.21.15.100:0/1116917918 learned_addr learned my addr 172.21.15.100:0/1116917918 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:55:16.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f78237fe640 10 monclient(hunting): get_auth_request con 0x7f782406eeb0 auth_method 0 2024-09-13T11:55:16.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f78237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:16.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f78237fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:16.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f78237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:16.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f78237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:16.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f78237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:16.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.723+0000 7f78237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:16.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f78237fe640 10 monclient(hunting): handle_auth_done global_id 5014 payload 274 2024-09-13T11:55:16.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f78237fe640 1 -- 172.21.15.100:0/1116917918 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78240a37d0 msgr2=0x7f78240a3bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:55:16.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f78237fe640 1 --2- 172.21.15.100:0/1116917918 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78240a37d0 0x7f78240a3bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f78237fe640 1 -- 172.21.15.100:0/1116917918 >> v1:172.21.15.122:6789/0 conn(0x7f7824137970 legacy=0x7f7824137e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:16.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f78237fe640 10 monclient: _finish_hunting 0 2024-09-13T11:55:16.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f78237fe640 1 monclient: found mon.noname-c 2024-09-13T11:55:16.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f78237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:16.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f78237fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.122:3300/0 2024-09-13T11:55:16.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f78237fe640 1 -- 172.21.15.100:0/1116917918 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78240a5240 con 0x7f782406eeb0 2024-09-13T11:55:16.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f7822ffd640 1 -- 172.21.15.100:0/1116917918 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1384648738 0 0) 0x7f782406e950 con 0x7f7824137970 2024-09-13T11:55:16.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f7822ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:55:16.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f78237fe640 1 --2- 172.21.15.100:0/1116917918 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f782406eeb0 0x7f78240a40f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f780c009870 tx=0x7f780c02a140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4458fd254b76284 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:16.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f7822ffd640 1 -- 172.21.15.100:0/1116917918 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c037070 con 0x7f782406eeb0 2024-09-13T11:55:16.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f7822ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:16.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f7822ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:55:16.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f7822ffd640 10 monclient: dump: 2024-09-13T11:55:16.737 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:16.737 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:16.737 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.738 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.738 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:16.738 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:16.738 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:16.738 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:16.738 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:16.738 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:16.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f7822ffd640 10 monclient: _finish_auth 0 2024-09-13T11:55:16.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f7822ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:55:16.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f7822ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:16.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f7822ffd640 1 -- 172.21.15.100:0/1116917918 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f780c032440 con 0x7f782406eeb0 2024-09-13T11:55:16.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f782a9bb640 5 monclient: authenticate success, global_id 5014 2024-09-13T11:55:16.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f782a9bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:55:16.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f7822ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:55:16.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f782a9bb640 10 monclient: get_monmap_and_config success 2024-09-13T11:55:16.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f782a9bb640 4 set_mon_vals no callback set 2024-09-13T11:55:16.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f7822ffd640 1 -- 172.21.15.100:0/1116917918 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c032810 con 0x7f782406eeb0 2024-09-13T11:55:16.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f7822ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:16.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f7822ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:55:16.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.724+0000 7f7822ffd640 10 monclient: dump: 2024-09-13T11:55:16.739 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:16.739 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:16.739 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.739 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.739 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:16.739 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:16.739 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:16.740 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:16.740 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:16.740 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:16.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.725+0000 7f782a9bb640 10 monclient: shutdown 2024-09-13T11:55:16.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.725+0000 7f782a9bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:55:16.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.725+0000 7f782a9bb640 1 -- 172.21.15.100:0/1116917918 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f782406eeb0 msgr2=0x7f78240a40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:16.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.725+0000 7f782a9bb640 1 --2- 172.21.15.100:0/1116917918 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f782406eeb0 0x7f78240a40f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f780c009870 tx=0x7f780c02a140 comp rx=0 tx=0).stop 2024-09-13T11:55:16.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.725+0000 7f782a9bb640 1 -- 172.21.15.100:0/1116917918 shutdown_connections 2024-09-13T11:55:16.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.725+0000 7f782a9bb640 1 --2- 172.21.15.100:0/1116917918 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f782406eeb0 0x7f78240a40f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.725+0000 7f782a9bb640 1 --2- 172.21.15.100:0/1116917918 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78240a37d0 0x7f78240a3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.725+0000 7f782a9bb640 1 -- 172.21.15.100:0/1116917918 >> 172.21.15.100:0/1116917918 conn(0x7f78240a83f0 msgr2=0x7f78240a48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:55:16.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.725+0000 7f782a9bb640 1 -- 172.21.15.100:0/1116917918 shutdown_connections 2024-09-13T11:55:16.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.725+0000 7f782a9bb640 1 -- 172.21.15.100:0/1116917918 wait complete. 2024-09-13T11:55:16.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.726+0000 7f782a9bb640 10 monclient: build_initial_monmap 2024-09-13T11:55:16.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.726+0000 7f782a9bb640 10 monclient: monmap: 2024-09-13T11:55:16.741 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:55:16.741 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:55:16.741 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:55:16.741 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:55:16.741 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:55:16.741 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:16.741 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:55:16.741 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:55:16.741 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:55:16.741 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:16.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.726+0000 7f782a9bb640 1 Processor -- start 2024-09-13T11:55:16.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.726+0000 7f782a9bb640 1 -- start start 2024-09-13T11:55:16.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.726+0000 7f782a9bb640 10 monclient: init 2024-09-13T11:55:16.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.726+0000 7f782a9bb640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:16.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.726+0000 7f782a9bb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:55:16.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f782a9bb640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f782406eeb0 0x7f78241ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:16.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f782a9bb640 10 monclient(hunting): picked mon.noname-c con 0x7f782406eeb0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:55:16.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f782a9bb640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78240a37d0 0x7f78241cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:16.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f782a9bb640 10 monclient(hunting): picked mon.noname-a con 0x7f78240a37d0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:55:16.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f782a9bb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7824137970 0x7f78241243b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:16.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f782a9bb640 10 monclient(hunting): picked mon.noname-b con 0x7f7824137970 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:55:16.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f782a9bb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:16.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f782a9bb640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f7824138530 con 0x7f78240a37d0 2024-09-13T11:55:16.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f782a9bb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:16.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f782a9bb640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f78240a4630 con 0x7f7824137970 2024-09-13T11:55:16.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f782a9bb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:16.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f782a9bb640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f78241386b0 con 0x7f782406eeb0 2024-09-13T11:55:16.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f782a9bb640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:16.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f782a9bb640 10 monclient(hunting): authenticate will time out at 897.320054s 2024-09-13T11:55:16.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f7823fff640 1 -- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f782406eeb0 msgr2=0x7f78241ccc70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:3301/0 2024-09-13T11:55:16.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f7823fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f782406eeb0 0x7f78241ccc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:55:16.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f78237fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78240a37d0 0x7f78241cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:16.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f78237fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78240a37d0 0x7f78241cd1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:43944/0 (socket says 172.21.15.100:43944) 2024-09-13T11:55:16.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f78237fe640 1 -- 172.21.15.100:0/3307533670 learned_addr learned my addr 172.21.15.100:0/3307533670 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:55:16.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f78237fe640 10 monclient(hunting): get_auth_request con 0x7f78240a37d0 auth_method 0 2024-09-13T11:55:16.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f78237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:16.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f78237fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:16.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f78237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:16.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f7828f58640 1 --2- 172.21.15.100:0/3307533670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7824137970 0x7f78241243b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:16.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f78237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:16.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f78237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:16.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f78237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:16.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f7828f58640 10 monclient(hunting): get_auth_request con 0x7f7824137970 auth_method 0 2024-09-13T11:55:16.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f7828f58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:16.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f7828f58640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:16.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f7828f58640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:16.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f7828f58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:16.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f7828f58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:16.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f7828f58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:16.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f78237fe640 10 monclient(hunting): handle_auth_done global_id 14226 payload 824 2024-09-13T11:55:16.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f78237fe640 1 -- 172.21.15.100:0/3307533670 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f782406eeb0 msgr2=0x7f78241ccc70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:55:16.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f78237fe640 1 --2- 172.21.15.100:0/3307533670 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f782406eeb0 0x7f78241ccc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f78237fe640 1 -- 172.21.15.100:0/3307533670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7824137970 msgr2=0x7f78241243b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:16.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f78237fe640 1 --2- 172.21.15.100:0/3307533670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7824137970 0x7f78241243b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f78237fe640 10 monclient: _finish_hunting 0 2024-09-13T11:55:16.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f78237fe640 1 monclient: found mon.noname-a 2024-09-13T11:55:16.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f78237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:16.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f78237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:55:16.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f78237fe640 1 -- 172.21.15.100:0/3307533670 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7824128190 con 0x7f78240a37d0 2024-09-13T11:55:16.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f78237fe640 1 --2- 172.21.15.100:0/3307533670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78240a37d0 0x7f78241cd1b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f780c0090b0 tx=0x7f780c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:16.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f7820ff9640 1 -- 172.21.15.100:0/3307533670 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c037040 con 0x7f78240a37d0 2024-09-13T11:55:16.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f7820ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:16.746 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f7820ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:55:16.746 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f7820ff9640 10 monclient: dump: 2024-09-13T11:55:16.746 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:16.746 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:16.746 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.746 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.746 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:16.746 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:16.746 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:16.746 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:16.746 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:16.746 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:16.746 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f7820ff9640 10 monclient: _finish_auth 0 2024-09-13T11:55:16.746 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f7820ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:55:16.747 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f7820ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:16.747 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f782a9bb640 5 monclient: authenticate success, global_id 14226 2024-09-13T11:55:16.747 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f7820ff9640 1 -- 172.21.15.100:0/3307533670 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f780c004230 con 0x7f78240a37d0 2024-09-13T11:55:16.747 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f7820ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:55:16.747 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f7820ff9640 1 -- 172.21.15.100:0/3307533670 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c03c730 con 0x7f78240a37d0 2024-09-13T11:55:16.747 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f782a9bb640 10 monclient: _renew_subs 2024-09-13T11:55:16.747 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f782a9bb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:16.747 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f782a9bb640 1 -- 172.21.15.100:0/3307533670 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78240a1750 con 0x7f78240a37d0 2024-09-13T11:55:16.747 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f7820ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:16.747 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f7820ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:16.747 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f7820ff9640 10 monclient: dump: 2024-09-13T11:55:16.747 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:16.747 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:16.748 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.748 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.748 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:16.748 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:16.748 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:16.748 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:16.748 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:16.748 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:16.748 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f78227fc640 4 set_mon_vals no callback set 2024-09-13T11:55:16.748 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f782a9bb640 10 monclient: _renew_subs 2024-09-13T11:55:16.749 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f782a9bb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:16.749 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.727+0000 7f782a9bb640 1 -- 172.21.15.100:0/3307533670 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7824137750 con 0x7f78240a37d0 2024-09-13T11:55:16.749 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.728+0000 7f782a9bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:16.749 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.728+0000 7f7820ff9640 1 -- 172.21.15.100:0/3307533670 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f780c03a070 con 0x7f78240a37d0 2024-09-13T11:55:16.749 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.728+0000 7f782a9bb640 1 --2- 172.21.15.100:0/3307533670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78241248f0 0x7f7824124cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:16.749 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.728+0000 7f782a9bb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:16.749 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.728+0000 7f782a9bb640 1 -- 172.21.15.100:0/3307533670 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f78240a7310 con 0x7f78241248f0 2024-09-13T11:55:16.749 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.728+0000 7f7823fff640 1 --2- 172.21.15.100:0/3307533670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78241248f0 0x7f7824124cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:16.749 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.728+0000 7f7823fff640 10 monclient: get_auth_request con 0x7f78241248f0 auth_method 0 2024-09-13T11:55:16.749 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.728+0000 7f7823fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:16.749 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.728+0000 7f7823fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:16.749 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.728+0000 7f7823fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:16.749 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.728+0000 7f7823fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:16.750 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.728+0000 7f7823fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:16.750 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.728+0000 7f7823fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:16.750 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.728+0000 7f7820ff9640 1 --2- 172.21.15.100:0/3307533670 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f780403df50 0x7f7804040410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:16.750 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.729+0000 7f7820ff9640 1 -- 172.21.15.100:0/3307533670 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4346+0+0 (secure 0 0 0) 0x7f780c072ef0 con 0x7f78240a37d0 2024-09-13T11:55:16.750 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.729+0000 7f7823fff640 10 monclient(hunting): handle_auth_done global_id 14229 payload 824 2024-09-13T11:55:16.750 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.729+0000 7f7823fff640 1 -- 172.21.15.100:0/3307533670 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7824137e20 con 0x7f78241248f0 2024-09-13T11:55:16.750 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.729+0000 7f7828f58640 1 --2- 172.21.15.100:0/3307533670 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f780403df50 0x7f7804040410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:16.750 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.729+0000 7f7828f58640 10 monclient: get_auth_request con 0x7f780403df50 auth_method 0 2024-09-13T11:55:16.750 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.729+0000 7f7823fff640 1 --2- 172.21.15.100:0/3307533670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78241248f0 0x7f7824124cd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f781400b450 tx=0x7f781400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:16.751 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.729+0000 7f7820ff9640 1 -- 172.21.15.100:0/3307533670 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7814003dc0 con 0x7f78241248f0 2024-09-13T11:55:16.751 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.729+0000 7f7820ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:16.751 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.729+0000 7f7820ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:16.751 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.729+0000 7f7820ff9640 10 monclient: dump: 2024-09-13T11:55:16.751 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:16.751 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:16.751 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.751 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.751 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:16.751 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:16.751 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:16.751 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:16.751 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:16.751 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:16.752 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.729+0000 7f7820ff9640 1 -- 172.21.15.100:0/3307533670 <== mon.0 v2:172.21.15.100:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7814015070 con 0x7f78241248f0 2024-09-13T11:55:16.752 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.729+0000 7f7820ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:16.752 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.729+0000 7f7820ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:55:16.752 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.729+0000 7f7820ff9640 1 -- 172.21.15.100:0/3307533670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78241248f0 msgr2=0x7f7824124cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:16.752 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.729+0000 7f7820ff9640 1 --2- 172.21.15.100:0/3307533670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78241248f0 0x7f7824124cd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f781400b450 tx=0x7f781400b910 comp rx=0 tx=0).stop 2024-09-13T11:55:16.752 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.729+0000 7f7820ff9640 1 -- 172.21.15.100:0/3307533670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78241248f0 msgr2=0x7f7824124cd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:55:16.752 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.729+0000 7f7820ff9640 1 --2- 172.21.15.100:0/3307533670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78241248f0 0x7f7824124cd0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.752 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.729+0000 7f7828f58640 1 --2- 172.21.15.100:0/3307533670 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f780403df50 0x7f7804040410 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f782406fc30 tx=0x7f781c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:16.754 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.735+0000 7f782a9bb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:55:16.754 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.735+0000 7f782a9bb640 1 --2- 172.21.15.100:0/3307533670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7824125210 0x7f7824070f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:16.754 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.735+0000 7f782a9bb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:16.754 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.735+0000 7f782a9bb640 1 -- 172.21.15.100:0/3307533670 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f7824128b00 con 0x7f7824125210 2024-09-13T11:55:16.754 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.735+0000 7f7823fff640 1 --2- 172.21.15.100:0/3307533670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7824125210 0x7f7824070f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:16.754 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.735+0000 7f7823fff640 10 monclient: get_auth_request con 0x7f7824125210 auth_method 0 2024-09-13T11:55:16.754 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.735+0000 7f7823fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:16.754 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.735+0000 7f7823fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:16.754 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.735+0000 7f7823fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:16.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.735+0000 7f7823fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:16.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.735+0000 7f7823fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:16.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.735+0000 7f7823fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:16.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.735+0000 7f7823fff640 10 monclient(hunting): handle_auth_done global_id 14232 payload 824 2024-09-13T11:55:16.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.735+0000 7f7823fff640 1 -- 172.21.15.100:0/3307533670 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7824126990 con 0x7f7824125210 2024-09-13T11:55:16.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.736+0000 7f7823fff640 1 --2- 172.21.15.100:0/3307533670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7824125210 0x7f7824070f00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f781400b450 tx=0x7f78140059d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:16.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.736+0000 7f7820ff9640 1 -- 172.21.15.100:0/3307533670 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7814023410 con 0x7f7824125210 2024-09-13T11:55:16.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.736+0000 7f7820ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:16.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.736+0000 7f7820ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:16.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.736+0000 7f7820ff9640 10 monclient: dump: 2024-09-13T11:55:16.755 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:16.755 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:16.755 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.756 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.756 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:16.756 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:16.756 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:16.756 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:16.756 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:16.756 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:16.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.736+0000 7f7820ff9640 1 -- 172.21.15.100:0/3307533670 <== mon.0 v2:172.21.15.100:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4710 (secure 0 0 0) 0x7f7814010400 con 0x7f7824125210 2024-09-13T11:55:16.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.736+0000 7f7820ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:55:16.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.736+0000 7f7820ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:55:16.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.736+0000 7f7820ff9640 1 -- 172.21.15.100:0/3307533670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7824125210 msgr2=0x7f7824070f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:16.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.736+0000 7f7820ff9640 1 --2- 172.21.15.100:0/3307533670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7824125210 0x7f7824070f00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f781400b450 tx=0x7f78140059d0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.736+0000 7f7820ff9640 1 -- 172.21.15.100:0/3307533670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7824125210 msgr2=0x7f7824070f00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:55:16.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.736+0000 7f7820ff9640 1 --2- 172.21.15.100:0/3307533670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7824125210 0x7f7824070f00 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f781400b450 tx=0x7f78140059d0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.757 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:55:16.757 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:55:16.757 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:55:16.757 INFO:teuthology.orchestra.run.smithi100.stdout: "state": "leader", 2024-09-13T11:55:16.757 INFO:teuthology.orchestra.run.smithi100.stdout: "election_epoch": 20, 2024-09-13T11:55:16.757 INFO:teuthology.orchestra.run.smithi100.stdout: "uptime": 17736, 2024-09-13T11:55:16.757 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum": [ 2024-09-13T11:55:16.757 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-13T11:55:16.757 INFO:teuthology.orchestra.run.smithi100.stdout: 1 2024-09-13T11:55:16.757 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:16.758 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_age": 0, 2024-09-13T11:55:16.758 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:55:16.758 INFO:teuthology.orchestra.run.smithi100.stdout: "required_con": "2450521708007866372", 2024-09-13T11:55:16.758 INFO:teuthology.orchestra.run.smithi100.stdout: "required_mon": [ 2024-09-13T11:55:16.758 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:55:16.758 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:55:16.758 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:55:16.758 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:55:16.758 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:55:16.758 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:55:16.758 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:55:16.758 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:55:16.758 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:55:16.759 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:55:16.759 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:55:16.759 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:16.759 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:55:16.759 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_mon": [ 2024-09-13T11:55:16.759 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:55:16.759 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:55:16.759 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:55:16.759 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:55:16.759 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:55:16.759 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:55:16.759 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:55:16.759 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:55:16.759 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:55:16.760 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:55:16.760 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:55:16.760 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:16.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:16.760 INFO:teuthology.orchestra.run.smithi100.stdout: "outside_quorum": [], 2024-09-13T11:55:16.760 INFO:teuthology.orchestra.run.smithi100.stdout: "extra_probe_peers": [], 2024-09-13T11:55:16.760 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_provider": [], 2024-09-13T11:55:16.760 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": { 2024-09-13T11:55:16.760 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-09-13T11:55:16.760 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "0b1f4b4d-fad4-4dd3-86ac-a286d80846c0", 2024-09-13T11:55:16.760 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:55:16.760 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:55:16.761 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release": 19, 2024-09-13T11:55:16.761 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release_name": "squid", 2024-09-13T11:55:16.761 INFO:teuthology.orchestra.run.smithi100.stdout: "election_strategy": 1, 2024-09-13T11:55:16.761 INFO:teuthology.orchestra.run.smithi100.stdout: "disallowed_leaders: ": "", 2024-09-13T11:55:16.761 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false, 2024-09-13T11:55:16.761 INFO:teuthology.orchestra.run.smithi100.stdout: "tiebreaker_mon": "", 2024-09-13T11:55:16.761 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_ranks: ": "", 2024-09-13T11:55:16.761 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:55:16.761 INFO:teuthology.orchestra.run.smithi100.stdout: "persistent": [ 2024-09-13T11:55:16.761 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:55:16.761 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:55:16.761 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:55:16.761 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:55:16.761 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:55:16.762 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:55:16.762 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:55:16.762 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:55:16.762 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:55:16.762 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:55:16.762 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:55:16.762 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:16.762 INFO:teuthology.orchestra.run.smithi100.stdout: "optional": [] 2024-09-13T11:55:16.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:16.762 INFO:teuthology.orchestra.run.smithi100.stdout: "mons": [ 2024-09-13T11:55:16.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.762 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:55:16.762 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:55:16.762 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:55:16.763 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:55:16.763 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.763 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:55:16.763 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3300", 2024-09-13T11:55:16.763 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:16.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:16.763 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.763 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:55:16.763 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789", 2024-09-13T11:55:16.763 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:16.763 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:16.763 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:16.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:16.764 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789/0", 2024-09-13T11:55:16.764 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-13T11:55:16.764 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:55:16.764 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:55:16.764 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:55:16.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:16.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.764 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:55:16.764 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:55:16.764 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:55:16.764 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:55:16.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.764 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:55:16.764 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:3300", 2024-09-13T11:55:16.765 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:16.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:16.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.765 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:55:16.765 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789", 2024-09-13T11:55:16.765 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:16.765 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:16.765 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:16.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:16.765 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789/0", 2024-09-13T11:55:16.765 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.122:6789/0", 2024-09-13T11:55:16.765 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:55:16.765 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:55:16.766 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:55:16.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:16.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.766 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:55:16.766 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:55:16.766 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:55:16.766 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:55:16.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.766 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:55:16.766 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3301", 2024-09-13T11:55:16.766 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:16.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:16.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.766 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:55:16.767 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790", 2024-09-13T11:55:16.767 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:16.767 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:16.767 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:16.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:16.767 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790/0", 2024-09-13T11:55:16.767 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-13T11:55:16.767 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:55:16.767 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:55:16.767 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:55:16.767 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:16.767 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:16.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:16.767 INFO:teuthology.orchestra.run.smithi100.stdout: "feature_map": { 2024-09-13T11:55:16.768 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": [ 2024-09-13T11:55:16.768 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.768 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:55:16.768 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:55:16.768 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:55:16.768 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:16.768 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:16.768 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": [ 2024-09-13T11:55:16.768 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.768 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:55:16.768 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:55:16.768 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 2 2024-09-13T11:55:16.768 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:16.769 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:16.769 INFO:teuthology.orchestra.run.smithi100.stdout: "client": [ 2024-09-13T11:55:16.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.769 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:55:16.769 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:55:16.769 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 3 2024-09-13T11:55:16.769 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:16.769 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:16.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:16.769 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false 2024-09-13T11:55:16.769 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:55:16.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.736+0000 7f782a9bb640 1 -- 172.21.15.100:0/3307533670 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f780403df50 msgr2=0x7f7804040410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:16.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.736+0000 7f782a9bb640 1 --2- 172.21.15.100:0/3307533670 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f780403df50 0x7f7804040410 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f782406fc30 tx=0x7f781c009040 comp rx=0 tx=0).stop 2024-09-13T11:55:16.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.736+0000 7f782a9bb640 10 monclient: shutdown 2024-09-13T11:55:16.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.736+0000 7f782a9bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:55:16.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.736+0000 7f782a9bb640 1 -- 172.21.15.100:0/3307533670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78240a37d0 msgr2=0x7f78241cd1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:16.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.736+0000 7f782a9bb640 1 --2- 172.21.15.100:0/3307533670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78240a37d0 0x7f78241cd1b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f780c0090b0 tx=0x7f780c0046c0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.736+0000 7f7828f58640 1 -- 172.21.15.100:0/3307533670 reap_dead start 2024-09-13T11:55:16.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.736+0000 7f7828f58640 1 -- 172.21.15.100:0/3307533670 reap_dead start 2024-09-13T11:55:16.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.736+0000 7f782a9bb640 1 -- 172.21.15.100:0/3307533670 shutdown_connections 2024-09-13T11:55:16.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.736+0000 7f782a9bb640 1 -- 172.21.15.100:0/3307533670 >> 172.21.15.100:0/3307533670 conn(0x7f78240a83f0 msgr2=0x7f782409da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:55:16.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.736+0000 7f782a9bb640 1 -- 172.21.15.100:0/3307533670 shutdown_connections 2024-09-13T11:55:16.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.736+0000 7f782a9bb640 1 -- 172.21.15.100:0/3307533670 wait complete. 2024-09-13T11:55:16.772 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-13T11:55:16.772 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-13T11:55:16.772 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-13T11:55:16.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.868+0000 7f3555dd0640 10 monclient: get_monmap_and_config 2024-09-13T11:55:16.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.868+0000 7f3555dd0640 10 monclient: build_initial_monmap 2024-09-13T11:55:16.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.868+0000 7f3555dd0640 10 monclient: monmap: 2024-09-13T11:55:16.870 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:55:16.870 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:16.870 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:55:16.869348+0000 2024-09-13T11:55:16.870 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:55:16.869348+0000 2024-09-13T11:55:16.870 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:55:16.870 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:16.871 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:55:16.871 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:55:16.871 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:55:16.871 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:55:16.871 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:55:16.871 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:16.881 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f3555dd0640 1 Processor -- start 2024-09-13T11:55:16.882 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f3555dd0640 1 -- start start 2024-09-13T11:55:16.882 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f3555dd0640 10 monclient: init 2024-09-13T11:55:16.882 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f3555dd0640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:16.882 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f3555dd0640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-13T11:55:16.882 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f3555dd0640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f35500a15a0 0x7f35500a1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:16.882 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f3555dd0640 10 monclient(hunting): picked mon.noname-a con 0x7f35500a15a0 addr v2:172.21.15.100:3300/0 2024-09-13T11:55:16.882 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f3555dd0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f355009f6d0 addr v1:172.21.15.122:6789/0 2024-09-13T11:55:16.882 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f3555dd0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f35500a1ec0 addr v1:172.21.15.100:6789/0 2024-09-13T11:55:16.882 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f3555dd0640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f355009ecf0 con 0x7f35500a1ec0 2024-09-13T11:55:16.882 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f3555dd0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35501351f0 con 0x7f355009f6d0 2024-09-13T11:55:16.883 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f3555dd0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:16.883 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f3555dd0640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f355009d3b0 con 0x7f35500a15a0 2024-09-13T11:55:16.883 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f3555dd0640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:16.883 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f3555dd0640 10 monclient(hunting): authenticate will time out at 897.462751s 2024-09-13T11:55:16.883 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f354f7fe640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f35500a15a0 0x7f35500a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:16.883 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f354ffff640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f35500a1ec0 0x7f3550134ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:42008/0 (socket says 172.21.15.100:42008) 2024-09-13T11:55:16.883 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f354f7fe640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f35500a15a0 0x7f35500a1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:43966/0 (socket says 172.21.15.100:43966) 2024-09-13T11:55:16.883 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f354ffff640 1 -- 172.21.15.100:0/483859464 learned_addr learned my addr 172.21.15.100:0/483859464 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:55:16.883 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f354f7fe640 10 monclient(hunting): get_auth_request con 0x7f35500a15a0 auth_method 0 2024-09-13T11:55:16.883 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f354f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:16.883 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f354f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:16.884 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f354f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:16.884 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f354f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:16.884 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.869+0000 7f354f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:16.884 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.870+0000 7f354f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:16.884 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.870+0000 7f354f7fe640 10 monclient(hunting): handle_auth_done global_id 14235 payload 274 2024-09-13T11:55:16.884 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.870+0000 7f354f7fe640 1 -- 172.21.15.100:0/483859464 >> v1:172.21.15.122:6789/0 conn(0x7f355009f6d0 legacy=0x7f35501343d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:16.884 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.870+0000 7f354f7fe640 1 -- 172.21.15.100:0/483859464 >> v1:172.21.15.100:6789/0 conn(0x7f35500a1ec0 legacy=0x7f3550134ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:16.884 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.870+0000 7f354f7fe640 10 monclient: _finish_hunting 0 2024-09-13T11:55:16.884 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.870+0000 7f354f7fe640 1 monclient: found mon.noname-a 2024-09-13T11:55:16.884 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.870+0000 7f354f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:16.885 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.870+0000 7f354f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:55:16.885 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.870+0000 7f354f7fe640 1 -- 172.21.15.100:0/483859464 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f355009ea10 con 0x7f35500a15a0 2024-09-13T11:55:16.885 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.870+0000 7f354e7fc640 1 -- 172.21.15.100:0/483859464 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2147191851 0 0) 0x7f355009ecf0 con 0x7f35500a1ec0 2024-09-13T11:55:16.885 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.870+0000 7f354e7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:55:16.885 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.870+0000 7f354f7fe640 1 --2- 172.21.15.100:0/483859464 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f35500a15a0 0x7f35500a1980 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f354000ba40 tx=0x7f354002c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e93fd5f716f6cb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:16.885 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.870+0000 7f354e7fc640 1 -- 172.21.15.100:0/483859464 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354003a070 con 0x7f35500a15a0 2024-09-13T11:55:16.885 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.870+0000 7f354e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:16.885 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.870+0000 7f354e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:55:16.885 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.870+0000 7f354e7fc640 10 monclient: dump: 2024-09-13T11:55:16.885 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:16.885 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:16.885 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.886 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.886 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:16.886 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:16.886 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:16.886 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:16.886 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:16.886 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:16.886 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.870+0000 7f354e7fc640 10 monclient: _finish_auth 0 2024-09-13T11:55:16.886 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.870+0000 7f354e7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:55:16.886 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.870+0000 7f354e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:16.887 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.870+0000 7f354e7fc640 1 -- 172.21.15.100:0/483859464 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3540031070 con 0x7f35500a15a0 2024-09-13T11:55:16.887 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.870+0000 7f3555dd0640 5 monclient: authenticate success, global_id 14235 2024-09-13T11:55:16.887 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.871+0000 7f3555dd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:55:16.887 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.871+0000 7f354e7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:55:16.887 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.871+0000 7f354e7fc640 1 -- 172.21.15.100:0/483859464 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35400328f0 con 0x7f35500a15a0 2024-09-13T11:55:16.887 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.871+0000 7f3555dd0640 10 monclient: get_monmap_and_config success 2024-09-13T11:55:16.887 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.871+0000 7f3555dd0640 4 set_mon_vals no callback set 2024-09-13T11:55:16.887 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.871+0000 7f354e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:16.887 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.871+0000 7f354e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:16.887 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.871+0000 7f354e7fc640 10 monclient: dump: 2024-09-13T11:55:16.888 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:16.888 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:16.888 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.888 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.888 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:16.888 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:16.888 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:16.888 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:16.888 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:16.888 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:16.888 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.871+0000 7f3555dd0640 10 monclient: shutdown 2024-09-13T11:55:16.889 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.871+0000 7f3555dd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:55:16.889 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.871+0000 7f3555dd0640 1 -- 172.21.15.100:0/483859464 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f35500a15a0 msgr2=0x7f35500a1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:16.889 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.871+0000 7f3555dd0640 1 --2- 172.21.15.100:0/483859464 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f35500a15a0 0x7f35500a1980 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f354000ba40 tx=0x7f354002c7d0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.889 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.871+0000 7f3555dd0640 1 -- 172.21.15.100:0/483859464 shutdown_connections 2024-09-13T11:55:16.889 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.871+0000 7f3555dd0640 1 --2- 172.21.15.100:0/483859464 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f35500a15a0 0x7f35500a1980 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.889 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.871+0000 7f3555dd0640 1 -- 172.21.15.100:0/483859464 >> 172.21.15.100:0/483859464 conn(0x7f35500a6470 msgr2=0x7f35500a2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:55:16.889 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.871+0000 7f3555dd0640 1 -- 172.21.15.100:0/483859464 shutdown_connections 2024-09-13T11:55:16.889 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.871+0000 7f3555dd0640 1 -- 172.21.15.100:0/483859464 wait complete. 2024-09-13T11:55:16.889 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.872+0000 7f3555dd0640 10 monclient: build_initial_monmap 2024-09-13T11:55:16.889 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.872+0000 7f3555dd0640 10 monclient: monmap: 2024-09-13T11:55:16.889 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:55:16.890 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:55:16.890 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:55:16.890 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:55:16.890 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:55:16.890 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:16.890 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:55:16.890 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:55:16.890 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:55:16.890 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:16.890 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.872+0000 7f3555dd0640 1 Processor -- start 2024-09-13T11:55:16.890 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.872+0000 7f3555dd0640 1 -- start start 2024-09-13T11:55:16.891 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.872+0000 7f3555dd0640 10 monclient: init 2024-09-13T11:55:16.891 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3555dd0640 10 monclient: _reopen_session rank -1 2024-09-13T11:55:16.891 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3555dd0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:55:16.891 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3555dd0640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f355009f6d0 0x7f35501cc700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:16.891 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3555dd0640 10 monclient(hunting): picked mon.noname-c con 0x7f355009f6d0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:55:16.891 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3555dd0640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f35500a15a0 0x7f35501ccc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:16.891 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3555dd0640 10 monclient(hunting): picked mon.noname-a con 0x7f35500a15a0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:55:16.891 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3555dd0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35500a1ec0 0x7f3550122e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:16.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3555dd0640 10 monclient(hunting): picked mon.noname-b con 0x7f35500a1ec0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:55:16.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3555dd0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:16.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3555dd0640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f35500a33d0 con 0x7f35500a15a0 2024-09-13T11:55:16.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3555dd0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:16.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3555dd0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f35500a3250 con 0x7f35500a1ec0 2024-09-13T11:55:16.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3555dd0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:16.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3555dd0640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f355009cfa0 con 0x7f355009f6d0 2024-09-13T11:55:16.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3555dd0640 10 monclient(hunting): _renew_subs 2024-09-13T11:55:16.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354f7fe640 1 -- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f355009f6d0 msgr2=0x7f35501cc700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:3301/0 2024-09-13T11:55:16.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3555dd0640 10 monclient(hunting): authenticate will time out at 897.466126s 2024-09-13T11:55:16.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354f7fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f355009f6d0 0x7f35501cc700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:55:16.893 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354effd640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f35500a15a0 0x7f35501ccc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:16.893 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354effd640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f35500a15a0 0x7f35501ccc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:43972/0 (socket says 172.21.15.100:43972) 2024-09-13T11:55:16.893 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354effd640 1 -- 172.21.15.100:0/4006084519 learned_addr learned my addr 172.21.15.100:0/4006084519 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:55:16.893 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354effd640 10 monclient(hunting): get_auth_request con 0x7f35500a15a0 auth_method 0 2024-09-13T11:55:16.893 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:16.893 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:16.893 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:16.893 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:16.893 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:16.893 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:16.894 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354ffff640 1 --2- 172.21.15.100:0/4006084519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35500a1ec0 0x7f3550122e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:16.894 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354ffff640 10 monclient(hunting): get_auth_request con 0x7f35500a1ec0 auth_method 0 2024-09-13T11:55:16.894 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:16.894 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:16.894 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:16.894 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:16.894 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:16.894 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:16.894 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354effd640 10 monclient(hunting): handle_auth_done global_id 14241 payload 824 2024-09-13T11:55:16.894 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354effd640 1 -- 172.21.15.100:0/4006084519 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f355009f6d0 msgr2=0x7f35501cc700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:55:16.894 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354effd640 1 --2- 172.21.15.100:0/4006084519 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f355009f6d0 0x7f35501cc700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.895 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354effd640 1 -- 172.21.15.100:0/4006084519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35500a1ec0 msgr2=0x7f3550122e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:16.895 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354effd640 1 --2- 172.21.15.100:0/4006084519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35500a1ec0 0x7f3550122e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.895 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354effd640 10 monclient: _finish_hunting 0 2024-09-13T11:55:16.895 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354effd640 1 monclient: found mon.noname-a 2024-09-13T11:55:16.895 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:55:16.895 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:55:16.895 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354effd640 1 -- 172.21.15.100:0/4006084519 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3550076a90 con 0x7f35500a15a0 2024-09-13T11:55:16.895 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354effd640 1 --2- 172.21.15.100:0/4006084519 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f35500a15a0 0x7f35501ccc40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f354400c400 tx=0x7f354400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:16.895 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3537fff640 1 -- 172.21.15.100:0/4006084519 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3544017070 con 0x7f35500a15a0 2024-09-13T11:55:16.895 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3537fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:16.895 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3537fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:55:16.896 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3537fff640 10 monclient: dump: 2024-09-13T11:55:16.896 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:16.896 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:16.896 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.896 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.896 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:16.896 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:16.896 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:16.896 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:16.896 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:16.897 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:16.897 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3537fff640 10 monclient: _finish_auth 0 2024-09-13T11:55:16.897 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3537fff640 10 monclient: _check_auth_tickets 2024-09-13T11:55:16.897 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3537fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:55:16.897 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3555dd0640 5 monclient: authenticate success, global_id 14241 2024-09-13T11:55:16.897 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3537fff640 1 -- 172.21.15.100:0/4006084519 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35440052d0 con 0x7f35500a15a0 2024-09-13T11:55:16.897 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3555dd0640 10 monclient: _renew_subs 2024-09-13T11:55:16.897 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3555dd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:16.897 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3555dd0640 1 -- 172.21.15.100:0/4006084519 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35501248e0 con 0x7f35500a15a0 2024-09-13T11:55:16.897 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3537fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:55:16.897 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3537fff640 1 -- 172.21.15.100:0/4006084519 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3544003830 con 0x7f35500a15a0 2024-09-13T11:55:16.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3537fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:16.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3537fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:55:16.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3537fff640 10 monclient: dump: 2024-09-13T11:55:16.898 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:16.898 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:16.898 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.898 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.898 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:16.898 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:16.898 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:16.899 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:16.899 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:16.899 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:16.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3555dd0640 10 monclient: _renew_subs 2024-09-13T11:55:16.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f354dffb640 4 set_mon_vals no callback set 2024-09-13T11:55:16.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3555dd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:55:16.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.873+0000 7f3555dd0640 1 -- 172.21.15.100:0/4006084519 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3550125440 con 0x7f35500a15a0 2024-09-13T11:55:16.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.874+0000 7f3555dd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:16.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.874+0000 7f3537fff640 1 -- 172.21.15.100:0/4006084519 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f354401a070 con 0x7f35500a15a0 2024-09-13T11:55:16.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.874+0000 7f3555dd0640 1 --2- 172.21.15.100:0/4006084519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3550123360 0x7f3550123740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:16.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.874+0000 7f3555dd0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:16.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.874+0000 7f3555dd0640 1 -- 172.21.15.100:0/4006084519 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f35500a1980 con 0x7f3550123360 2024-09-13T11:55:16.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.874+0000 7f354f7fe640 1 --2- 172.21.15.100:0/4006084519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3550123360 0x7f3550123740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:16.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.874+0000 7f3537fff640 1 --2- 172.21.15.100:0/4006084519 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f353c03df80 0x7f353c040440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:16.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.874+0000 7f354f7fe640 10 monclient: get_auth_request con 0x7f3550123360 auth_method 0 2024-09-13T11:55:16.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.874+0000 7f354f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:16.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.874+0000 7f354f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:16.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.874+0000 7f354f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:16.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.874+0000 7f3537fff640 1 -- 172.21.15.100:0/4006084519 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4346+0+0 (secure 0 0 0) 0x7f35440518a0 con 0x7f35500a15a0 2024-09-13T11:55:16.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.874+0000 7f354f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:16.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.874+0000 7f354f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:16.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.874+0000 7f354f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:16.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.875+0000 7f354ffff640 1 --2- 172.21.15.100:0/4006084519 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f353c03df80 0x7f353c040440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:16.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.875+0000 7f354ffff640 10 monclient: get_auth_request con 0x7f353c03df80 auth_method 0 2024-09-13T11:55:16.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.875+0000 7f354f7fe640 10 monclient(hunting): handle_auth_done global_id 5029 payload 824 2024-09-13T11:55:16.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.875+0000 7f354f7fe640 1 -- 172.21.15.100:0/4006084519 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f35500008d0 con 0x7f3550123360 2024-09-13T11:55:16.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.875+0000 7f354f7fe640 1 --2- 172.21.15.100:0/4006084519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3550123360 0x7f3550123740 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f35500a0160 tx=0x7f3540004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:16.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.875+0000 7f354ffff640 1 --2- 172.21.15.100:0/4006084519 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f353c03df80 0x7f353c040440 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f35480012e0 tx=0x7f3548004d40 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:16.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.875+0000 7f3537fff640 1 -- 172.21.15.100:0/4006084519 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354003a040 con 0x7f3550123360 2024-09-13T11:55:16.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.875+0000 7f3537fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:16.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.875+0000 7f3537fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:55:16.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.875+0000 7f3537fff640 10 monclient: dump: 2024-09-13T11:55:16.901 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:16.902 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:16.902 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.902 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.902 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:16.902 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:16.902 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:16.902 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:16.902 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:16.902 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:16.902 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.875+0000 7f3537fff640 1 -- 172.21.15.100:0/4006084519 <== mon.1 v2:172.21.15.122:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3540004220 con 0x7f3550123360 2024-09-13T11:55:16.902 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.875+0000 7f3537fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:55:16.903 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.875+0000 7f3537fff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:55:16.903 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.875+0000 7f3537fff640 1 -- 172.21.15.100:0/4006084519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3550123360 msgr2=0x7f3550123740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:16.903 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.875+0000 7f3537fff640 1 --2- 172.21.15.100:0/4006084519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3550123360 0x7f3550123740 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f35500a0160 tx=0x7f3540004730 comp rx=0 tx=0).stop 2024-09-13T11:55:16.903 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.875+0000 7f3537fff640 1 -- 172.21.15.100:0/4006084519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3550123360 msgr2=0x7f3550123740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:55:16.903 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.875+0000 7f3537fff640 1 --2- 172.21.15.100:0/4006084519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3550123360 0x7f3550123740 secure :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f35500a0160 tx=0x7f3540004730 comp rx=0 tx=0).stop 2024-09-13T11:55:16.905 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.881+0000 7f3555dd0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:55:16.905 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.881+0000 7f3555dd0640 1 --2- 172.21.15.100:0/4006084519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3550123c80 0x7f3550071490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:55:16.905 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.881+0000 7f3555dd0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:55:16.905 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.881+0000 7f3555dd0640 1 -- 172.21.15.100:0/4006084519 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f35501cd180 con 0x7f3550123c80 2024-09-13T11:55:16.905 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.882+0000 7f354f7fe640 1 --2- 172.21.15.100:0/4006084519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3550123c80 0x7f3550071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:55:16.905 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.882+0000 7f354f7fe640 10 monclient: get_auth_request con 0x7f3550123c80 auth_method 0 2024-09-13T11:55:16.906 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.882+0000 7f354f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:55:16.906 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.882+0000 7f354f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:55:16.906 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.882+0000 7f354f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:55:16.906 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.882+0000 7f354f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:55:16.906 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.882+0000 7f354f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:55:16.906 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.882+0000 7f354f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:55:16.906 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.882+0000 7f354f7fe640 10 monclient(hunting): handle_auth_done global_id 5032 payload 824 2024-09-13T11:55:16.906 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.882+0000 7f354f7fe640 1 -- 172.21.15.100:0/4006084519 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3550127600 con 0x7f3550123c80 2024-09-13T11:55:16.906 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.882+0000 7f354f7fe640 1 --2- 172.21.15.100:0/4006084519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3550123c80 0x7f3550071490 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f354000ba40 tx=0x7f35400046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:55:16.906 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.882+0000 7f3537fff640 1 -- 172.21.15.100:0/4006084519 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354004a070 con 0x7f3550123c80 2024-09-13T11:55:16.907 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.882+0000 7f3537fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:55:16.907 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.882+0000 7f3537fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:55:16.907 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.882+0000 7f3537fff640 10 monclient: dump: 2024-09-13T11:55:16.907 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:55:16.907 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:55:16.907 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.907 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:55:16.907 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:55:16.907 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:55:16.907 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:55:16.907 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:55:16.908 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:55:16.908 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:55:16.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.883+0000 7f3537fff640 1 -- 172.21.15.100:0/4006084519 <== mon.1 v2:172.21.15.122:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f3540046810 con 0x7f3550123c80 2024-09-13T11:55:16.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.883+0000 7f3537fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:55:16.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.883+0000 7f3537fff640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:55:16.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.883+0000 7f3537fff640 1 -- 172.21.15.100:0/4006084519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3550123c80 msgr2=0x7f3550071490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:16.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.883+0000 7f3537fff640 1 --2- 172.21.15.100:0/4006084519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3550123c80 0x7f3550071490 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f354000ba40 tx=0x7f35400046c0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.883+0000 7f3537fff640 1 -- 172.21.15.100:0/4006084519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3550123c80 msgr2=0x7f3550071490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:55:16.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.883+0000 7f3537fff640 1 --2- 172.21.15.100:0/4006084519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3550123c80 0x7f3550071490 secure :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f354000ba40 tx=0x7f35400046c0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.909 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:55:16.909 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:55:16.909 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:55:16.909 INFO:teuthology.orchestra.run.smithi100.stdout: "state": "peon", 2024-09-13T11:55:16.909 INFO:teuthology.orchestra.run.smithi100.stdout: "election_epoch": 20, 2024-09-13T11:55:16.909 INFO:teuthology.orchestra.run.smithi100.stdout: "uptime": 155913, 2024-09-13T11:55:16.909 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum": [ 2024-09-13T11:55:16.909 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-13T11:55:16.909 INFO:teuthology.orchestra.run.smithi100.stdout: 1 2024-09-13T11:55:16.910 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:16.910 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_age": 0, 2024-09-13T11:55:16.910 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:55:16.910 INFO:teuthology.orchestra.run.smithi100.stdout: "required_con": "2450521708007866372", 2024-09-13T11:55:16.910 INFO:teuthology.orchestra.run.smithi100.stdout: "required_mon": [ 2024-09-13T11:55:16.910 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:55:16.910 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:55:16.910 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:55:16.910 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:55:16.910 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:55:16.911 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:55:16.911 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:55:16.911 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:55:16.911 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:55:16.911 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:55:16.911 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:55:16.911 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:16.911 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:55:16.911 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_mon": [ 2024-09-13T11:55:16.911 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:55:16.911 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:55:16.912 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:55:16.912 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:55:16.912 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:55:16.912 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:55:16.912 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:55:16.912 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:55:16.912 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:55:16.912 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:55:16.912 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:55:16.912 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:16.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:16.913 INFO:teuthology.orchestra.run.smithi100.stdout: "outside_quorum": [], 2024-09-13T11:55:16.913 INFO:teuthology.orchestra.run.smithi100.stdout: "extra_probe_peers": [], 2024-09-13T11:55:16.913 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_provider": [], 2024-09-13T11:55:16.913 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": { 2024-09-13T11:55:16.913 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-09-13T11:55:16.913 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "0b1f4b4d-fad4-4dd3-86ac-a286d80846c0", 2024-09-13T11:55:16.913 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:55:16.913 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:55:16.913 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release": 19, 2024-09-13T11:55:16.913 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release_name": "squid", 2024-09-13T11:55:16.913 INFO:teuthology.orchestra.run.smithi100.stdout: "election_strategy": 1, 2024-09-13T11:55:16.914 INFO:teuthology.orchestra.run.smithi100.stdout: "disallowed_leaders: ": "", 2024-09-13T11:55:16.914 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false, 2024-09-13T11:55:16.914 INFO:teuthology.orchestra.run.smithi100.stdout: "tiebreaker_mon": "", 2024-09-13T11:55:16.914 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_ranks: ": "", 2024-09-13T11:55:16.914 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:55:16.914 INFO:teuthology.orchestra.run.smithi100.stdout: "persistent": [ 2024-09-13T11:55:16.914 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:55:16.914 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:55:16.914 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:55:16.914 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:55:16.914 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:55:16.915 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:55:16.915 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:55:16.915 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:55:16.915 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:55:16.915 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:55:16.915 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:55:16.915 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:16.915 INFO:teuthology.orchestra.run.smithi100.stdout: "optional": [] 2024-09-13T11:55:16.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:16.915 INFO:teuthology.orchestra.run.smithi100.stdout: "mons": [ 2024-09-13T11:55:16.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.916 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:55:16.916 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:55:16.916 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:55:16.916 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:55:16.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.916 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:55:16.916 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3300", 2024-09-13T11:55:16.916 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:16.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:16.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.917 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:55:16.917 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789", 2024-09-13T11:55:16.917 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:16.917 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:16.917 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:16.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:16.917 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789/0", 2024-09-13T11:55:16.917 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-13T11:55:16.917 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:55:16.917 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:55:16.917 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:55:16.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:16.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.918 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:55:16.918 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:55:16.918 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:55:16.918 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:55:16.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.918 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:55:16.918 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:3300", 2024-09-13T11:55:16.918 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:16.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:16.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.919 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:55:16.919 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789", 2024-09-13T11:55:16.919 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:16.919 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:16.919 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:16.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:16.919 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789/0", 2024-09-13T11:55:16.919 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.122:6789/0", 2024-09-13T11:55:16.919 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:55:16.919 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:55:16.920 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:55:16.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:16.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.920 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:55:16.920 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:55:16.920 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:55:16.920 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:55:16.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.920 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:55:16.920 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3301", 2024-09-13T11:55:16.921 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:16.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:16.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.921 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:55:16.921 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790", 2024-09-13T11:55:16.921 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:55:16.921 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:16.921 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:16.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:16.921 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790/0", 2024-09-13T11:55:16.922 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-13T11:55:16.922 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:55:16.922 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:55:16.922 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:55:16.922 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:16.922 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:16.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:16.922 INFO:teuthology.orchestra.run.smithi100.stdout: "feature_map": { 2024-09-13T11:55:16.922 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": [ 2024-09-13T11:55:16.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.922 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:55:16.923 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:55:16.923 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:55:16.923 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:16.923 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:16.923 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": [ 2024-09-13T11:55:16.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.923 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:55:16.923 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:55:16.923 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 4 2024-09-13T11:55:16.923 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:16.923 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:16.924 INFO:teuthology.orchestra.run.smithi100.stdout: "client": [ 2024-09-13T11:55:16.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.924 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:55:16.924 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:55:16.924 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 6 2024-09-13T11:55:16.924 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:16.924 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:55:16.924 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": [ 2024-09-13T11:55:16.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:55:16.924 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:55:16.924 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:55:16.925 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:55:16.925 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:55:16.925 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:55:16.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:55:16.925 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false 2024-09-13T11:55:16.925 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:55:16.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.883+0000 7f3555dd0640 1 -- 172.21.15.100:0/4006084519 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f353c03df80 msgr2=0x7f353c040440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:16.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.883+0000 7f3555dd0640 1 --2- 172.21.15.100:0/4006084519 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f353c03df80 0x7f353c040440 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f35480012e0 tx=0x7f3548004d40 comp rx=0 tx=0).stop 2024-09-13T11:55:16.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.883+0000 7f3555dd0640 10 monclient: shutdown 2024-09-13T11:55:16.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.883+0000 7f3555dd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:55:16.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.883+0000 7f3555dd0640 1 -- 172.21.15.100:0/4006084519 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f35500a15a0 msgr2=0x7f35501ccc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:55:16.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.883+0000 7f3555dd0640 1 --2- 172.21.15.100:0/4006084519 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f35500a15a0 0x7f35501ccc40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f354400c400 tx=0x7f354400c8c0 comp rx=0 tx=0).stop 2024-09-13T11:55:16.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.883+0000 7f354ffff640 1 -- 172.21.15.100:0/4006084519 reap_dead start 2024-09-13T11:55:16.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.883+0000 7f354ffff640 1 -- 172.21.15.100:0/4006084519 reap_dead start 2024-09-13T11:55:16.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.883+0000 7f3555dd0640 1 -- 172.21.15.100:0/4006084519 shutdown_connections 2024-09-13T11:55:16.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.883+0000 7f3555dd0640 1 -- 172.21.15.100:0/4006084519 >> 172.21.15.100:0/4006084519 conn(0x7f35500a6470 msgr2=0x7f35500a52d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:55:16.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.883+0000 7f3555dd0640 1 -- 172.21.15.100:0/4006084519 shutdown_connections 2024-09-13T11:55:16.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:55:16.883+0000 7f3555dd0640 1 -- 172.21.15.100:0/4006084519 wait complete. 2024-09-13T11:55:16.928 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-13T11:55:16.928 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-13T11:55:18.442 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36264]: starting. 2024-09-13T11:55:18.443 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36264]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-36263 2024-09-13T11:55:18.443 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36264]: created object 0... 2024-09-13T11:55:18.443 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36264]: created object 25... 2024-09-13T11:55:18.443 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36264]: created object 49... 2024-09-13T11:55:18.443 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36264]: finishing. 2024-09-13T11:55:18.443 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36264]: shutting down. 2024-09-13T11:55:19.361 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36265]: starting. 2024-09-13T11:55:19.362 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36265]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-36263 2024-09-13T11:55:19.362 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36265]: shutting down. 2024-09-13T11:55:21.455 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:21.456 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36298]: starting. 2024-09-13T11:55:21.456 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36298]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-36263 2024-09-13T11:55:21.456 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36298]: created object 0... 2024-09-13T11:55:21.456 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36298]: created object 25... 2024-09-13T11:55:21.456 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36298]: created object 49... 2024-09-13T11:55:21.456 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36298]: finishing. 2024-09-13T11:55:21.456 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36298]: shutting down. 2024-09-13T11:55:21.467 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:21.467 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36300]: starting. 2024-09-13T11:55:21.467 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36300]: listing objects. 2024-09-13T11:55:21.467 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36300]: listed object 0... 2024-09-13T11:55:21.468 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36300]: listed object 25... 2024-09-13T11:55:21.468 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36300]: saw 50 objects 2024-09-13T11:55:21.468 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36300]: shutting down. 2024-09-13T11:55:22.379 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:22.379 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36299]: starting. 2024-09-13T11:55:22.379 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36299]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-36263 2024-09-13T11:55:22.379 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36299]: shutting down. 2024-09-13T11:55:22.380 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:22.380 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:22.380 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:55:22.383 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:55:24.094 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36367]: starting. 2024-09-13T11:55:24.094 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36367]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-36366 2024-09-13T11:55:24.095 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36367]: created object 0... 2024-09-13T11:55:24.095 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36367]: created object 25... 2024-09-13T11:55:24.095 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36367]: created object 49... 2024-09-13T11:55:24.095 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36367]: finishing. 2024-09-13T11:55:24.095 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36367]: shutting down. 2024-09-13T11:55:25.018 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36368]: starting. 2024-09-13T11:55:25.018 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36368]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-36366 2024-09-13T11:55:25.018 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36368]: shutting down. 2024-09-13T11:55:27.101 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:27.101 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36401]: starting. 2024-09-13T11:55:27.101 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36401]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-36366 2024-09-13T11:55:27.101 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36401]: created object 0... 2024-09-13T11:55:27.101 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36401]: created object 25... 2024-09-13T11:55:27.101 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36401]: created object 49... 2024-09-13T11:55:27.101 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36401]: finishing. 2024-09-13T11:55:27.102 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36401]: shutting down. 2024-09-13T11:55:27.113 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:27.113 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36403]: starting. 2024-09-13T11:55:27.113 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36403]: listing objects. 2024-09-13T11:55:27.113 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36403]: listed object 0... 2024-09-13T11:55:27.113 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36403]: listed object 25... 2024-09-13T11:55:27.113 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36403]: saw 50 objects 2024-09-13T11:55:27.113 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36403]: shutting down. 2024-09-13T11:55:28.030 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:28.030 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36402]: starting. 2024-09-13T11:55:28.030 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36402]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-36366 2024-09-13T11:55:28.030 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36402]: shutting down. 2024-09-13T11:55:28.031 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:28.031 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:28.031 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:55:28.034 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:55:30.126 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36470]: starting. 2024-09-13T11:55:30.127 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36470]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-36469 2024-09-13T11:55:30.127 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36470]: created object 0... 2024-09-13T11:55:30.127 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36470]: created object 25... 2024-09-13T11:55:30.127 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36470]: created object 49... 2024-09-13T11:55:30.127 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36470]: finishing. 2024-09-13T11:55:30.127 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36470]: shutting down. 2024-09-13T11:55:31.043 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36471]: starting. 2024-09-13T11:55:31.043 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36471]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-36469 2024-09-13T11:55:31.043 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36471]: shutting down. 2024-09-13T11:55:33.138 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:33.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36504]: starting. 2024-09-13T11:55:33.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36504]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-36469 2024-09-13T11:55:33.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36504]: created object 0... 2024-09-13T11:55:33.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36504]: created object 25... 2024-09-13T11:55:33.139 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36504]: created object 49... 2024-09-13T11:55:33.139 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36504]: finishing. 2024-09-13T11:55:33.139 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36504]: shutting down. 2024-09-13T11:55:33.151 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:33.151 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36506]: starting. 2024-09-13T11:55:33.151 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36506]: listing objects. 2024-09-13T11:55:33.151 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36506]: listed object 0... 2024-09-13T11:55:33.151 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36506]: listed object 25... 2024-09-13T11:55:33.151 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36506]: saw 50 objects 2024-09-13T11:55:33.151 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36506]: shutting down. 2024-09-13T11:55:34.016 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:34.016 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36505]: starting. 2024-09-13T11:55:34.016 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36505]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-36469 2024-09-13T11:55:34.016 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36505]: shutting down. 2024-09-13T11:55:34.017 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:34.017 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:34.017 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:55:34.021 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:55:36.166 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36573]: starting. 2024-09-13T11:55:36.166 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36573]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-36572 2024-09-13T11:55:36.167 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36573]: created object 0... 2024-09-13T11:55:36.167 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36573]: created object 25... 2024-09-13T11:55:36.167 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36573]: created object 49... 2024-09-13T11:55:36.167 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36573]: finishing. 2024-09-13T11:55:36.167 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36573]: shutting down. 2024-09-13T11:55:37.077 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36574]: starting. 2024-09-13T11:55:37.077 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36574]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-36572 2024-09-13T11:55:37.077 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36574]: shutting down. 2024-09-13T11:55:39.178 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:39.178 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36607]: starting. 2024-09-13T11:55:39.178 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36607]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-36572 2024-09-13T11:55:39.178 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36607]: created object 0... 2024-09-13T11:55:39.178 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36607]: created object 25... 2024-09-13T11:55:39.178 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36607]: created object 49... 2024-09-13T11:55:39.178 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36607]: finishing. 2024-09-13T11:55:39.178 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36607]: shutting down. 2024-09-13T11:55:39.190 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:39.190 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36609]: starting. 2024-09-13T11:55:39.190 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36609]: listing objects. 2024-09-13T11:55:39.190 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36609]: listed object 0... 2024-09-13T11:55:39.190 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36609]: listed object 25... 2024-09-13T11:55:39.190 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36609]: saw 50 objects 2024-09-13T11:55:39.190 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36609]: shutting down. 2024-09-13T11:55:40.101 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:40.101 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36608]: starting. 2024-09-13T11:55:40.101 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36608]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-36572 2024-09-13T11:55:40.102 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36608]: shutting down. 2024-09-13T11:55:40.103 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:40.103 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:40.103 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:55:40.106 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:55:42.190 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36676]: starting. 2024-09-13T11:55:42.190 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36676]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-36675 2024-09-13T11:55:42.190 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36676]: created object 0... 2024-09-13T11:55:42.190 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36676]: created object 25... 2024-09-13T11:55:42.190 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36676]: created object 49... 2024-09-13T11:55:42.190 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36676]: finishing. 2024-09-13T11:55:42.191 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36676]: shutting down. 2024-09-13T11:55:43.113 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36677]: starting. 2024-09-13T11:55:43.114 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36677]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-36675 2024-09-13T11:55:43.114 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36677]: shutting down. 2024-09-13T11:55:45.104 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:45.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36710]: starting. 2024-09-13T11:55:45.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36710]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-36675 2024-09-13T11:55:45.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36710]: created object 0... 2024-09-13T11:55:45.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36710]: created object 25... 2024-09-13T11:55:45.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36710]: created object 49... 2024-09-13T11:55:45.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36710]: finishing. 2024-09-13T11:55:45.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36710]: shutting down. 2024-09-13T11:55:45.116 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:45.116 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36712]: starting. 2024-09-13T11:55:45.116 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36712]: listing objects. 2024-09-13T11:55:45.117 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36712]: listed object 0... 2024-09-13T11:55:45.117 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36712]: listed object 25... 2024-09-13T11:55:45.117 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36712]: saw 50 objects 2024-09-13T11:55:45.117 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36712]: shutting down. 2024-09-13T11:55:46.024 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:46.024 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36711]: starting. 2024-09-13T11:55:46.024 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36711]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-36675 2024-09-13T11:55:46.024 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36711]: shutting down. 2024-09-13T11:55:46.025 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:46.025 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:46.026 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:55:46.028 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:55:48.104 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36779]: starting. 2024-09-13T11:55:48.104 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36779]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-36778 2024-09-13T11:55:48.104 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36779]: created object 0... 2024-09-13T11:55:48.104 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36779]: created object 25... 2024-09-13T11:55:48.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36779]: created object 49... 2024-09-13T11:55:48.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36779]: finishing. 2024-09-13T11:55:48.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36779]: shutting down. 2024-09-13T11:55:49.032 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36780]: starting. 2024-09-13T11:55:49.032 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36780]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-36778 2024-09-13T11:55:49.032 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36780]: shutting down. 2024-09-13T11:55:51.324 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:51.324 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36813]: starting. 2024-09-13T11:55:51.324 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36813]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-36778 2024-09-13T11:55:51.324 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36813]: created object 0... 2024-09-13T11:55:51.324 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36813]: created object 25... 2024-09-13T11:55:51.324 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36813]: created object 49... 2024-09-13T11:55:51.324 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36813]: finishing. 2024-09-13T11:55:51.324 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36813]: shutting down. 2024-09-13T11:55:51.337 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:51.337 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36815]: starting. 2024-09-13T11:55:51.337 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36815]: listing objects. 2024-09-13T11:55:51.337 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36815]: listed object 0... 2024-09-13T11:55:51.337 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36815]: listed object 25... 2024-09-13T11:55:51.337 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36815]: saw 50 objects 2024-09-13T11:55:51.337 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36815]: shutting down. 2024-09-13T11:55:52.042 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:52.042 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36814]: starting. 2024-09-13T11:55:52.043 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36814]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-36778 2024-09-13T11:55:52.043 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36814]: shutting down. 2024-09-13T11:55:52.044 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:52.044 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:52.044 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:55:52.046 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:55:54.114 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36882]: starting. 2024-09-13T11:55:54.115 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36882]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-36881 2024-09-13T11:55:54.115 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36882]: created object 0... 2024-09-13T11:55:54.115 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36882]: created object 25... 2024-09-13T11:55:54.115 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36882]: created object 49... 2024-09-13T11:55:54.115 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36882]: finishing. 2024-09-13T11:55:54.115 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36882]: shutting down. 2024-09-13T11:55:55.027 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36883]: starting. 2024-09-13T11:55:55.028 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36883]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-36881 2024-09-13T11:55:55.028 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36883]: shutting down. 2024-09-13T11:55:57.111 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:57.112 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36916]: starting. 2024-09-13T11:55:57.112 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36916]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-36881 2024-09-13T11:55:57.112 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36916]: created object 0... 2024-09-13T11:55:57.112 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36916]: created object 25... 2024-09-13T11:55:57.112 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36916]: created object 49... 2024-09-13T11:55:57.112 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36916]: finishing. 2024-09-13T11:55:57.112 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[36916]: shutting down. 2024-09-13T11:55:57.124 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:57.124 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36918]: starting. 2024-09-13T11:55:57.124 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36918]: listing objects. 2024-09-13T11:55:57.124 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36918]: listed object 0... 2024-09-13T11:55:57.124 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36918]: listed object 25... 2024-09-13T11:55:57.124 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36918]: saw 50 objects 2024-09-13T11:55:57.124 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[36918]: shutting down. 2024-09-13T11:55:58.043 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:58.043 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36917]: starting. 2024-09-13T11:55:58.043 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36917]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-36881 2024-09-13T11:55:58.043 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[36917]: shutting down. 2024-09-13T11:55:58.044 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:58.045 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:55:58.045 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:55:58.047 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:56:00.129 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36985]: starting. 2024-09-13T11:56:00.129 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36985]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-36984 2024-09-13T11:56:00.129 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36985]: created object 0... 2024-09-13T11:56:00.129 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36985]: created object 25... 2024-09-13T11:56:00.129 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36985]: created object 49... 2024-09-13T11:56:00.129 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36985]: finishing. 2024-09-13T11:56:00.129 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[36985]: shutting down. 2024-09-13T11:56:01.053 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36986]: starting. 2024-09-13T11:56:01.054 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36986]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-36984 2024-09-13T11:56:01.054 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[36986]: shutting down. 2024-09-13T11:56:03.146 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:03.146 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37019]: starting. 2024-09-13T11:56:03.146 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37019]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-36984 2024-09-13T11:56:03.146 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37019]: created object 0... 2024-09-13T11:56:03.147 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37019]: created object 25... 2024-09-13T11:56:03.147 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37019]: created object 49... 2024-09-13T11:56:03.147 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37019]: finishing. 2024-09-13T11:56:03.147 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37019]: shutting down. 2024-09-13T11:56:03.158 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:03.158 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37021]: starting. 2024-09-13T11:56:03.159 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37021]: listing objects. 2024-09-13T11:56:03.159 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37021]: listed object 0... 2024-09-13T11:56:03.159 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37021]: listed object 25... 2024-09-13T11:56:03.159 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37021]: saw 50 objects 2024-09-13T11:56:03.159 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37021]: shutting down. 2024-09-13T11:56:04.022 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:04.022 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37020]: starting. 2024-09-13T11:56:04.023 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37020]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-36984 2024-09-13T11:56:04.023 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37020]: shutting down. 2024-09-13T11:56:04.024 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:04.024 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:04.024 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:56:04.027 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:56:06.163 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37088]: starting. 2024-09-13T11:56:06.163 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37088]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-37087 2024-09-13T11:56:06.163 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37088]: created object 0... 2024-09-13T11:56:06.163 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37088]: created object 25... 2024-09-13T11:56:06.163 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37088]: created object 49... 2024-09-13T11:56:06.164 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37088]: finishing. 2024-09-13T11:56:06.164 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37088]: shutting down. 2024-09-13T11:56:07.086 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37089]: starting. 2024-09-13T11:56:07.086 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37089]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-37087 2024-09-13T11:56:07.086 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37089]: shutting down. 2024-09-13T11:56:09.393 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:09.393 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37122]: starting. 2024-09-13T11:56:09.393 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37122]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-37087 2024-09-13T11:56:09.393 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37122]: created object 0... 2024-09-13T11:56:09.393 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37122]: created object 25... 2024-09-13T11:56:09.393 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37122]: created object 49... 2024-09-13T11:56:09.393 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37122]: finishing. 2024-09-13T11:56:09.393 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37122]: shutting down. 2024-09-13T11:56:09.408 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:09.409 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37124]: starting. 2024-09-13T11:56:09.409 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37124]: listing objects. 2024-09-13T11:56:09.409 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37124]: listed object 0... 2024-09-13T11:56:09.409 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37124]: listed object 25... 2024-09-13T11:56:09.409 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37124]: saw 50 objects 2024-09-13T11:56:09.409 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37124]: shutting down. 2024-09-13T11:56:10.105 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:10.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37123]: starting. 2024-09-13T11:56:10.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37123]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-37087 2024-09-13T11:56:10.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37123]: shutting down. 2024-09-13T11:56:10.106 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:10.106 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:10.106 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:56:10.109 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:56:12.197 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37191]: starting. 2024-09-13T11:56:12.197 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37191]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-37190 2024-09-13T11:56:12.197 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37191]: created object 0... 2024-09-13T11:56:12.197 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37191]: created object 25... 2024-09-13T11:56:12.198 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37191]: created object 49... 2024-09-13T11:56:12.198 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37191]: finishing. 2024-09-13T11:56:12.198 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37191]: shutting down. 2024-09-13T11:56:13.110 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37192]: starting. 2024-09-13T11:56:13.111 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37192]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-37190 2024-09-13T11:56:13.111 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37192]: shutting down. 2024-09-13T11:56:15.111 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:15.112 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37225]: starting. 2024-09-13T11:56:15.112 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37225]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-37190 2024-09-13T11:56:15.112 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37225]: created object 0... 2024-09-13T11:56:15.112 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37225]: created object 25... 2024-09-13T11:56:15.112 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37225]: created object 49... 2024-09-13T11:56:15.112 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37225]: finishing. 2024-09-13T11:56:15.112 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37225]: shutting down. 2024-09-13T11:56:15.123 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:15.123 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37227]: starting. 2024-09-13T11:56:15.123 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37227]: listing objects. 2024-09-13T11:56:15.123 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37227]: listed object 0... 2024-09-13T11:56:15.123 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37227]: listed object 25... 2024-09-13T11:56:15.123 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37227]: saw 50 objects 2024-09-13T11:56:15.123 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37227]: shutting down. 2024-09-13T11:56:16.029 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:16.029 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37226]: starting. 2024-09-13T11:56:16.029 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37226]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-37190 2024-09-13T11:56:16.029 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37226]: shutting down. 2024-09-13T11:56:16.030 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:16.030 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:16.031 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:56:16.033 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:56:18.112 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37294]: starting. 2024-09-13T11:56:18.113 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37294]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-37293 2024-09-13T11:56:18.113 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37294]: created object 0... 2024-09-13T11:56:18.113 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37294]: created object 25... 2024-09-13T11:56:18.113 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37294]: created object 49... 2024-09-13T11:56:18.113 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37294]: finishing. 2024-09-13T11:56:18.113 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37294]: shutting down. 2024-09-13T11:56:19.039 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37295]: starting. 2024-09-13T11:56:19.039 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37295]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-37293 2024-09-13T11:56:19.039 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37295]: shutting down. 2024-09-13T11:56:21.126 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:21.126 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37328]: starting. 2024-09-13T11:56:21.126 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37328]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-37293 2024-09-13T11:56:21.126 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37328]: created object 0... 2024-09-13T11:56:21.126 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37328]: created object 25... 2024-09-13T11:56:21.126 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37328]: created object 49... 2024-09-13T11:56:21.126 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37328]: finishing. 2024-09-13T11:56:21.126 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37328]: shutting down. 2024-09-13T11:56:21.138 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:21.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37330]: starting. 2024-09-13T11:56:21.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37330]: listing objects. 2024-09-13T11:56:21.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37330]: listed object 0... 2024-09-13T11:56:21.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37330]: listed object 25... 2024-09-13T11:56:21.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37330]: saw 50 objects 2024-09-13T11:56:21.139 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37330]: shutting down. 2024-09-13T11:56:22.055 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:22.055 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37329]: starting. 2024-09-13T11:56:22.055 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37329]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-37293 2024-09-13T11:56:22.055 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37329]: shutting down. 2024-09-13T11:56:22.056 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:22.056 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:22.056 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:56:22.059 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:56:24.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37397]: starting. 2024-09-13T11:56:24.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37397]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-37396 2024-09-13T11:56:24.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37397]: created object 0... 2024-09-13T11:56:24.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37397]: created object 25... 2024-09-13T11:56:24.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37397]: created object 49... 2024-09-13T11:56:24.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37397]: finishing. 2024-09-13T11:56:24.105 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37397]: shutting down. 2024-09-13T11:56:25.028 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37398]: starting. 2024-09-13T11:56:25.028 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37398]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-37396 2024-09-13T11:56:25.028 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37398]: shutting down. 2024-09-13T11:56:27.128 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:27.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37431]: starting. 2024-09-13T11:56:27.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37431]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-37396 2024-09-13T11:56:27.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37431]: created object 0... 2024-09-13T11:56:27.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37431]: created object 25... 2024-09-13T11:56:27.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37431]: created object 49... 2024-09-13T11:56:27.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37431]: finishing. 2024-09-13T11:56:27.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37431]: shutting down. 2024-09-13T11:56:27.140 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:27.140 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37433]: starting. 2024-09-13T11:56:27.140 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37433]: listing objects. 2024-09-13T11:56:27.140 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37433]: listed object 0... 2024-09-13T11:56:27.140 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37433]: listed object 25... 2024-09-13T11:56:27.140 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37433]: saw 50 objects 2024-09-13T11:56:27.140 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37433]: shutting down. 2024-09-13T11:56:28.046 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:28.046 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37432]: starting. 2024-09-13T11:56:28.046 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37432]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-37396 2024-09-13T11:56:28.046 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37432]: shutting down. 2024-09-13T11:56:28.047 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:28.047 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:28.048 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:56:28.050 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:56:30.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37500]: starting. 2024-09-13T11:56:30.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37500]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-37499 2024-09-13T11:56:30.131 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37500]: created object 0... 2024-09-13T11:56:30.131 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37500]: created object 25... 2024-09-13T11:56:30.131 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37500]: created object 49... 2024-09-13T11:56:30.131 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37500]: finishing. 2024-09-13T11:56:30.131 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37500]: shutting down. 2024-09-13T11:56:31.064 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37501]: starting. 2024-09-13T11:56:31.064 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37501]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-37499 2024-09-13T11:56:31.064 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37501]: shutting down. 2024-09-13T11:56:33.153 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:33.154 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37534]: starting. 2024-09-13T11:56:33.154 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37534]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-37499 2024-09-13T11:56:33.154 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37534]: created object 0... 2024-09-13T11:56:33.154 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37534]: created object 25... 2024-09-13T11:56:33.154 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37534]: created object 49... 2024-09-13T11:56:33.154 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37534]: finishing. 2024-09-13T11:56:33.154 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37534]: shutting down. 2024-09-13T11:56:33.166 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:33.166 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37536]: starting. 2024-09-13T11:56:33.166 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37536]: listing objects. 2024-09-13T11:56:33.166 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37536]: listed object 0... 2024-09-13T11:56:33.166 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37536]: listed object 25... 2024-09-13T11:56:33.166 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37536]: saw 50 objects 2024-09-13T11:56:33.166 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37536]: shutting down. 2024-09-13T11:56:34.029 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:34.030 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37535]: starting. 2024-09-13T11:56:34.030 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37535]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-37499 2024-09-13T11:56:34.030 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37535]: shutting down. 2024-09-13T11:56:34.031 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:34.031 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:34.031 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:56:34.034 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:56:36.167 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37603]: starting. 2024-09-13T11:56:36.167 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37603]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-37602 2024-09-13T11:56:36.167 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37603]: created object 0... 2024-09-13T11:56:36.167 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37603]: created object 25... 2024-09-13T11:56:36.167 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37603]: created object 49... 2024-09-13T11:56:36.167 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37603]: finishing. 2024-09-13T11:56:36.167 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37603]: shutting down. 2024-09-13T11:56:37.089 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37604]: starting. 2024-09-13T11:56:37.089 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37604]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-37602 2024-09-13T11:56:37.090 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37604]: shutting down. 2024-09-13T11:56:39.196 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:39.196 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37637]: starting. 2024-09-13T11:56:39.196 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37637]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-37602 2024-09-13T11:56:39.196 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37637]: created object 0... 2024-09-13T11:56:39.196 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37637]: created object 25... 2024-09-13T11:56:39.196 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37637]: created object 49... 2024-09-13T11:56:39.196 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37637]: finishing. 2024-09-13T11:56:39.196 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37637]: shutting down. 2024-09-13T11:56:39.208 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:39.208 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37639]: starting. 2024-09-13T11:56:39.208 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37639]: listing objects. 2024-09-13T11:56:39.208 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37639]: listed object 0... 2024-09-13T11:56:39.208 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37639]: listed object 25... 2024-09-13T11:56:39.208 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37639]: saw 50 objects 2024-09-13T11:56:39.208 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37639]: shutting down. 2024-09-13T11:56:40.115 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:40.115 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37638]: starting. 2024-09-13T11:56:40.115 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37638]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-37602 2024-09-13T11:56:40.115 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37638]: shutting down. 2024-09-13T11:56:40.117 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:40.117 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:40.117 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:56:40.119 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:56:42.213 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37706]: starting. 2024-09-13T11:56:42.213 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37706]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-37705 2024-09-13T11:56:42.213 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37706]: created object 0... 2024-09-13T11:56:42.213 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37706]: created object 25... 2024-09-13T11:56:42.213 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37706]: created object 49... 2024-09-13T11:56:42.213 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37706]: finishing. 2024-09-13T11:56:42.213 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37706]: shutting down. 2024-09-13T11:56:43.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37707]: starting. 2024-09-13T11:56:43.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37707]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-37705 2024-09-13T11:56:43.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37707]: shutting down. 2024-09-13T11:56:45.112 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:45.113 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37740]: starting. 2024-09-13T11:56:45.113 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37740]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-37705 2024-09-13T11:56:45.113 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37740]: created object 0... 2024-09-13T11:56:45.113 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37740]: created object 25... 2024-09-13T11:56:45.113 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37740]: created object 49... 2024-09-13T11:56:45.113 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37740]: finishing. 2024-09-13T11:56:45.113 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37740]: shutting down. 2024-09-13T11:56:45.125 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:45.125 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37742]: starting. 2024-09-13T11:56:45.125 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37742]: listing objects. 2024-09-13T11:56:45.125 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37742]: listed object 0... 2024-09-13T11:56:45.125 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37742]: listed object 25... 2024-09-13T11:56:45.125 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37742]: saw 50 objects 2024-09-13T11:56:45.125 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37742]: shutting down. 2024-09-13T11:56:46.033 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:46.034 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37741]: starting. 2024-09-13T11:56:46.034 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37741]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-37705 2024-09-13T11:56:46.034 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37741]: shutting down. 2024-09-13T11:56:46.035 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:46.035 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:46.035 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:56:46.038 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:56:46.929 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-13T11:56:46.929 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-13T11:56:46.929 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-13T11:56:46.933 INFO:tasks.ceph.mon.c:Started 2024-09-13T11:56:46.933 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-13T11:56:46.933 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:56:47.028 INFO:tasks.ceph.mon.c.smithi100.stderr:2024-09-13T11:56:47.027+0000 7f99033eec80 -1 mon.c@-1(???).log v0 log_external_backlog rewinding external_log_to from 115 -> 0 (sync_force? mon rebuild?) 2024-09-13T11:56:47.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.043+0000 7f97fec23640 10 monclient: get_monmap_and_config 2024-09-13T11:56:47.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.043+0000 7f97fec23640 10 monclient: build_initial_monmap 2024-09-13T11:56:47.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.043+0000 7f97fec23640 10 monclient: monmap: 2024-09-13T11:56:47.045 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:56:47.045 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:56:47.045 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:56:47.044769+0000 2024-09-13T11:56:47.045 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:56:47.044769+0000 2024-09-13T11:56:47.046 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:56:47.046 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:56:47.046 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:56:47.046 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:56:47.046 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:56:47.046 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:56:47.046 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:56:47.046 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:56:47.056 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.044+0000 7f97fec23640 1 Processor -- start 2024-09-13T11:56:47.056 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.044+0000 7f97fec23640 1 -- start start 2024-09-13T11:56:47.056 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.044+0000 7f97fec23640 10 monclient: init 2024-09-13T11:56:47.056 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fec23640 10 monclient: _reopen_session rank -1 2024-09-13T11:56:47.056 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fec23640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-13T11:56:47.056 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fec23640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f97f8137a60 0x7f97f80a2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:56:47.056 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fec23640 10 monclient(hunting): picked mon.noname-c con 0x7f97f8137a60 addr v2:172.21.15.122:3300/0 2024-09-13T11:56:47.056 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fec23640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f97f8137e40 addr v1:172.21.15.122:6789/0 2024-09-13T11:56:47.057 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fec23640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f97f8138250 0x7f97f80a5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:56:47.057 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fec23640 10 monclient(hunting): picked mon.noname-a con 0x7f97f8138250 addr v2:172.21.15.100:3300/0 2024-09-13T11:56:47.057 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fec23640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97f80a5be0 con 0x7f97f8137e40 2024-09-13T11:56:47.057 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fec23640 10 monclient(hunting): start opening mon connection 2024-09-13T11:56:47.057 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fec23640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f97f80a4410 con 0x7f97f8138250 2024-09-13T11:56:47.057 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fec23640 10 monclient(hunting): start opening mon connection 2024-09-13T11:56:47.057 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fec23640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f97f80a76b0 con 0x7f97f8137a60 2024-09-13T11:56:47.057 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fec23640 10 monclient(hunting): _renew_subs 2024-09-13T11:56:47.057 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fec23640 10 monclient(hunting): authenticate will time out at 987.638309s 2024-09-13T11:56:47.057 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fd1c0640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f97f8138250 0x7f97f80a5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:56:47.057 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fd1c0640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f97f8138250 0x7f97f80a5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:48218/0 (socket says 172.21.15.100:48218) 2024-09-13T11:56:47.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fd1c0640 1 -- 172.21.15.100:0/227664784 learned_addr learned my addr 172.21.15.100:0/227664784 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:56:47.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fd1c0640 10 monclient(hunting): get_auth_request con 0x7f97f8138250 auth_method 0 2024-09-13T11:56:47.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fd1c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:56:47.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fd1c0640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:56:47.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fd1c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:56:47.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fd1c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:56:47.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fd1c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:56:47.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fd1c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:56:47.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fc9bf640 1 --2- 172.21.15.100:0/227664784 >> v2:172.21.15.122:3300/0 conn(0x7f97f8137a60 0x7f97f80a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:56:47.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fc9bf640 10 monclient(hunting): get_auth_request con 0x7f97f8137a60 auth_method 0 2024-09-13T11:56:47.059 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fc9bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:56:47.059 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fc9bf640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:56:47.059 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fc9bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:56:47.059 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fd1c0640 10 monclient(hunting): handle_auth_done global_id 14739 payload 274 2024-09-13T11:56:47.059 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fd1c0640 1 -- 172.21.15.100:0/227664784 >> v2:172.21.15.122:3300/0 conn(0x7f97f8137a60 msgr2=0x7f97f80a2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:56:47.059 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fd1c0640 1 --2- 172.21.15.100:0/227664784 >> v2:172.21.15.122:3300/0 conn(0x7f97f8137a60 0x7f97f80a2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:47.059 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fd1c0640 1 -- 172.21.15.100:0/227664784 >> v1:172.21.15.122:6789/0 conn(0x7f97f8137e40 legacy=0x7f97f80a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:56:47.059 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fd1c0640 10 monclient: _finish_hunting 0 2024-09-13T11:56:47.059 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fd1c0640 1 monclient: found mon.noname-a 2024-09-13T11:56:47.059 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fd1c0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:56:47.059 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fd1c0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:56:47.059 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.045+0000 7f97fd1c0640 1 -- 172.21.15.100:0/227664784 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f80a19e0 con 0x7f97f8138250 2024-09-13T11:56:47.060 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.046+0000 7f97fd1c0640 1 --2- 172.21.15.100:0/227664784 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97f8138250 0x7f97f80a5060 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f97f80a5db0 tx=0x7f97f002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a18195d5ff289f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:56:47.060 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.046+0000 7f97f77fe640 1 -- 172.21.15.100:0/227664784 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f0036070 con 0x7f97f8138250 2024-09-13T11:56:47.060 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.046+0000 7f97f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:56:47.060 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.046+0000 7f97f77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:56:47.060 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.046+0000 7f97f77fe640 10 monclient: dump: 2024-09-13T11:56:47.060 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:56:47.060 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:56:47.060 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:47.060 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:47.060 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:56:47.060 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:56:47.060 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:56:47.060 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:56:47.061 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:56:47.061 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:56:47.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.046+0000 7f97f77fe640 10 monclient: _finish_auth 0 2024-09-13T11:56:47.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.046+0000 7f97f77fe640 10 monclient: _check_auth_tickets 2024-09-13T11:56:47.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.046+0000 7f97f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:56:47.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.046+0000 7f97fec23640 5 monclient: authenticate success, global_id 14739 2024-09-13T11:56:47.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.046+0000 7f97f77fe640 1 -- 172.21.15.100:0/227664784 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97f0031600 con 0x7f97f8138250 2024-09-13T11:56:47.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.046+0000 7f97fec23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:56:47.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.046+0000 7f97f77fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:56:47.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.046+0000 7f97fec23640 10 monclient: get_monmap_and_config success 2024-09-13T11:56:47.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.046+0000 7f97f77fe640 1 -- 172.21.15.100:0/227664784 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f00319d0 con 0x7f97f8138250 2024-09-13T11:56:47.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.046+0000 7f97f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:56:47.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.046+0000 7f97fec23640 4 set_mon_vals no callback set 2024-09-13T11:56:47.062 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.046+0000 7f97f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:56:47.062 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.046+0000 7f97f77fe640 10 monclient: dump: 2024-09-13T11:56:47.062 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:56:47.062 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:56:47.062 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:47.062 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:47.062 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:56:47.062 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:56:47.062 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:56:47.062 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:56:47.062 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:56:47.062 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:56:47.062 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.047+0000 7f97fec23640 10 monclient: shutdown 2024-09-13T11:56:47.062 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.047+0000 7f97fec23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:56:47.063 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.047+0000 7f97fec23640 1 -- 172.21.15.100:0/227664784 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97f8138250 msgr2=0x7f97f80a5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:56:47.063 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.047+0000 7f97fec23640 1 --2- 172.21.15.100:0/227664784 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97f8138250 0x7f97f80a5060 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f97f80a5db0 tx=0x7f97f002a610 comp rx=0 tx=0).stop 2024-09-13T11:56:47.063 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.047+0000 7f97fec23640 1 -- 172.21.15.100:0/227664784 shutdown_connections 2024-09-13T11:56:47.063 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.047+0000 7f97fec23640 1 --2- 172.21.15.100:0/227664784 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97f8138250 0x7f97f80a5060 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:47.063 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.047+0000 7f97fec23640 1 --2- 172.21.15.100:0/227664784 >> v2:172.21.15.122:3300/0 conn(0x7f97f8137a60 0x7f97f80a2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:47.063 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.047+0000 7f97fec23640 1 -- 172.21.15.100:0/227664784 >> 172.21.15.100:0/227664784 conn(0x7f97f80aa580 msgr2=0x7f97f80a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:56:47.063 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.047+0000 7f97fec23640 1 -- 172.21.15.100:0/227664784 shutdown_connections 2024-09-13T11:56:47.063 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.047+0000 7f97fec23640 1 -- 172.21.15.100:0/227664784 wait complete. 2024-09-13T11:56:47.063 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.047+0000 7f97fec23640 10 monclient: build_initial_monmap 2024-09-13T11:56:47.063 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.047+0000 7f97fec23640 10 monclient: monmap: 2024-09-13T11:56:47.063 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:56:47.063 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:56:47.064 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:56:47.064 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:56:47.064 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:56:47.064 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:56:47.064 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:56:47.064 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:56:47.064 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:56:47.064 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:56:47.064 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.048+0000 7f97fec23640 1 Processor -- start 2024-09-13T11:56:47.064 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.048+0000 7f97fec23640 1 -- start start 2024-09-13T11:56:47.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.048+0000 7f97fec23640 10 monclient: init 2024-09-13T11:56:47.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.048+0000 7f97fec23640 10 monclient: _reopen_session rank -1 2024-09-13T11:56:47.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.048+0000 7f97fec23640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:56:47.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.048+0000 7f97fec23640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97f8137a60 0x7f97f81cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:56:47.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.048+0000 7f97fec23640 10 monclient(hunting): picked mon.noname-b con 0x7f97f8137a60 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:56:47.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.048+0000 7f97fec23640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97f8137e40 0x7f97f81cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:56:47.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.048+0000 7f97fec23640 10 monclient(hunting): picked mon.noname-c con 0x7f97f8137e40 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:56:47.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.048+0000 7f97fec23640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97f8138250 0x7f97f8124380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:56:47.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.048+0000 7f97fec23640 10 monclient(hunting): picked mon.noname-a con 0x7f97f8138250 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:56:47.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.048+0000 7f97fec23640 10 monclient(hunting): start opening mon connection 2024-09-13T11:56:47.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.048+0000 7f97fec23640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f97f806ff00 con 0x7f97f8138250 2024-09-13T11:56:47.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.048+0000 7f97fec23640 10 monclient(hunting): start opening mon connection 2024-09-13T11:56:47.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.048+0000 7f97fec23640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f97f806fd80 con 0x7f97f8137a60 2024-09-13T11:56:47.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.048+0000 7f97fec23640 10 monclient(hunting): start opening mon connection 2024-09-13T11:56:47.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.048+0000 7f97fec23640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f97f80a55a0 con 0x7f97f8137e40 2024-09-13T11:56:47.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.048+0000 7f97fec23640 10 monclient(hunting): _renew_subs 2024-09-13T11:56:47.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.048+0000 7f97fec23640 10 monclient(hunting): authenticate will time out at 987.641909s 2024-09-13T11:56:47.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fd1c0640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97f8138250 0x7f97f8124380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:56:47.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97f7fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97f8137e40 0x7f97f81cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:56:47.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fd1c0640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97f8138250 0x7f97f8124380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:48232/0 (socket says 172.21.15.100:48232) 2024-09-13T11:56:47.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fd1c0640 1 -- 172.21.15.100:0/3674725326 learned_addr learned my addr 172.21.15.100:0/3674725326 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:56:47.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fd1c0640 10 monclient(hunting): get_auth_request con 0x7f97f8138250 auth_method 0 2024-09-13T11:56:47.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fd1c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:56:47.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fd1c0640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:56:47.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fd1c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:56:47.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fc9bf640 1 --2- 172.21.15.100:0/3674725326 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97f8137a60 0x7f97f81cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:56:47.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fc9bf640 10 monclient(hunting): get_auth_request con 0x7f97f8137a60 auth_method 0 2024-09-13T11:56:47.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fc9bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:56:47.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fc9bf640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:56:47.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fc9bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:56:47.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97f7fff640 10 monclient(hunting): get_auth_request con 0x7f97f8137e40 auth_method 0 2024-09-13T11:56:47.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:56:47.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:56:47.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:56:47.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fd1c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:56:47.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fd1c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:56:47.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fd1c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:56:47.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fc9bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:56:47.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fc9bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:56:47.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fc9bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:56:47.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97f7fff640 1 -- 172.21.15.100:0/3674725326 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97f8137e40 msgr2=0x7f97f81cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-13T11:56:47.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97f7fff640 1 -- 172.21.15.100:0/3674725326 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97f8137e40 msgr2=0x7f97f81cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T11:56:47.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97f7fff640 1 --2- 172.21.15.100:0/3674725326 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97f8137e40 0x7f97f81cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:56:47.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97f7fff640 1 --2- 172.21.15.100:0/3674725326 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97f8137e40 0x7f97f81cccd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:56:47.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fd1c0640 10 monclient(hunting): handle_auth_done global_id 14742 payload 824 2024-09-13T11:56:47.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fd1c0640 1 -- 172.21.15.100:0/3674725326 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97f8137e40 msgr2=0x7f97f81cccd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:56:47.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fd1c0640 1 --2- 172.21.15.100:0/3674725326 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97f8137e40 0x7f97f81cccd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:47.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fd1c0640 1 -- 172.21.15.100:0/3674725326 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97f8137a60 msgr2=0x7f97f81cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:56:47.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fd1c0640 1 --2- 172.21.15.100:0/3674725326 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97f8137a60 0x7f97f81cc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:47.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fd1c0640 10 monclient: _finish_hunting 0 2024-09-13T11:56:47.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fd1c0640 1 monclient: found mon.noname-a 2024-09-13T11:56:47.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fd1c0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:56:47.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fd1c0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:56:47.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fd1c0640 1 -- 172.21.15.100:0/3674725326 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f8125e40 con 0x7f97f8138250 2024-09-13T11:56:47.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fc9bf640 1 --2- 172.21.15.100:0/3674725326 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97f8137a60 0x7f97f81cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:56:47.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fd1c0640 1 --2- 172.21.15.100:0/3674725326 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97f8138250 0x7f97f8124380 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f97f0009f10 tx=0x7f97f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:56:47.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97f57fa640 1 -- 172.21.15.100:0/3674725326 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f0036050 con 0x7f97f8138250 2024-09-13T11:56:47.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97f57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:56:47.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97f57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:56:47.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97f57fa640 10 monclient: dump: 2024-09-13T11:56:47.069 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:56:47.069 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:56:47.069 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:47.069 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:47.069 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:56:47.070 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:56:47.070 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:56:47.070 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:56:47.070 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:56:47.070 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:56:47.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97f57fa640 10 monclient: _finish_auth 0 2024-09-13T11:56:47.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97f57fa640 10 monclient: _check_auth_tickets 2024-09-13T11:56:47.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:56:47.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97f57fa640 1 -- 172.21.15.100:0/3674725326 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97f0042070 con 0x7f97f8138250 2024-09-13T11:56:47.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fec23640 5 monclient: authenticate success, global_id 14742 2024-09-13T11:56:47.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fec23640 10 monclient: _renew_subs 2024-09-13T11:56:47.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fec23640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:56:47.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fec23640 1 -- 172.21.15.100:0/3674725326 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97f80a0870 con 0x7f97f8138250 2024-09-13T11:56:47.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97f57fa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:56:47.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97f57fa640 1 -- 172.21.15.100:0/3674725326 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f0007bd0 con 0x7f97f8138250 2024-09-13T11:56:47.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97f57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:56:47.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97f57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:56:47.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97f57fa640 10 monclient: dump: 2024-09-13T11:56:47.071 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:56:47.071 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:56:47.071 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:47.071 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:47.071 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:56:47.071 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:56:47.072 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:56:47.072 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:56:47.072 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:56:47.072 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:56:47.072 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97f6ffd640 4 set_mon_vals no callback set 2024-09-13T11:56:47.072 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fec23640 10 monclient: _renew_subs 2024-09-13T11:56:47.072 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fec23640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:56:47.072 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.049+0000 7f97fec23640 1 -- 172.21.15.100:0/3674725326 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97f81cd210 con 0x7f97f8138250 2024-09-13T11:56:47.072 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.050+0000 7f97f57fa640 1 -- 172.21.15.100:0/3674725326 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f97f0047460 con 0x7f97f8138250 2024-09-13T11:56:47.072 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.050+0000 7f97fec23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:56:47.072 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.050+0000 7f97fec23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:56:47.072 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.050+0000 7f97fec23640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:56:47.072 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.050+0000 7f97fec23640 1 -- 172.21.15.100:0/3674725326 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97f8127fa0 con 0x7f97f8138250 2024-09-13T11:56:47.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.050+0000 7f97f57fa640 1 --2- 172.21.15.100:0/3674725326 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f97dc03dfd0 0x7f97dc040490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:56:47.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.051+0000 7f97f57fa640 1 -- 172.21.15.100:0/3674725326 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(206..206 src has 1..206) ==== 4738+0+0 (secure 0 0 0) 0x7f97f0073730 con 0x7f97f8138250 2024-09-13T11:56:47.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.051+0000 7f97fc9bf640 1 --2- 172.21.15.100:0/3674725326 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f97dc03dfd0 0x7f97dc040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:56:47.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.051+0000 7f97fc9bf640 10 monclient: get_auth_request con 0x7f97dc03dfd0 auth_method 0 2024-09-13T11:56:47.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.051+0000 7f97fc9bf640 1 --2- 172.21.15.100:0/3674725326 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f97dc03dfd0 0x7f97dc040490 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f97ec006fd0 tx=0x7f97ec008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:56:47.074 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.055+0000 7f97f57fa640 1 -- 172.21.15.100:0/3674725326 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f97f0007e20 con 0x7f97f8138250 2024-09-13T11:56:47.074 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.055+0000 7f97f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:56:47.074 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.055+0000 7f97f57fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:56:47.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.298+0000 7f97fec23640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:56:47.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.298+0000 7f97fec23640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:56:47.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.298+0000 7f97fec23640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:56:47.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.298+0000 7f97fec23640 1 -- 172.21.15.100:0/3674725326 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f97f80a2980 con 0x7f97f8138250 2024-09-13T11:56:47.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.299+0000 7f97f57fa640 1 -- 172.21.15.100:0/3674725326 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f97f0047c70 con 0x7f97f8138250 2024-09-13T11:56:47.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.299+0000 7f97f57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:56:47.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.299+0000 7f97f57fa640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:56:47.301 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":20,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":90,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:56:47.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.301+0000 7f97fec23640 1 -- 172.21.15.100:0/3674725326 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f97dc03dfd0 msgr2=0x7f97dc040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:56:47.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.301+0000 7f97fec23640 1 --2- 172.21.15.100:0/3674725326 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f97dc03dfd0 0x7f97dc040490 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f97ec006fd0 tx=0x7f97ec008040 comp rx=0 tx=0).stop 2024-09-13T11:56:47.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.301+0000 7f97fec23640 10 monclient: shutdown 2024-09-13T11:56:47.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.301+0000 7f97fec23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:56:47.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.301+0000 7f97fec23640 1 -- 172.21.15.100:0/3674725326 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97f8138250 msgr2=0x7f97f8124380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:56:47.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.301+0000 7f97fec23640 1 --2- 172.21.15.100:0/3674725326 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97f8138250 0x7f97f8124380 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f97f0009f10 tx=0x7f97f0004a40 comp rx=0 tx=0).stop 2024-09-13T11:56:47.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.301+0000 7f97fec23640 1 -- 172.21.15.100:0/3674725326 shutdown_connections 2024-09-13T11:56:47.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.301+0000 7f97fec23640 1 --2- 172.21.15.100:0/3674725326 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f97dc03dfd0 0x7f97dc040490 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:47.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.301+0000 7f97fec23640 1 --2- 172.21.15.100:0/3674725326 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f97f8138250 0x7f97f8124380 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:47.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.301+0000 7f97fec23640 1 --2- 172.21.15.100:0/3674725326 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f97f8137e40 0x7f97f81cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:47.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.301+0000 7f97fec23640 1 --2- 172.21.15.100:0/3674725326 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97f8137a60 0x7f97f81cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:47.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.301+0000 7f97fec23640 1 -- 172.21.15.100:0/3674725326 >> 172.21.15.100:0/3674725326 conn(0x7f97f80aa580 msgr2=0x7f97f80a2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:56:47.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.301+0000 7f97fec23640 1 -- 172.21.15.100:0/3674725326 shutdown_connections 2024-09-13T11:56:47.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:47.301+0000 7f97fec23640 1 -- 172.21.15.100:0/3674725326 wait complete. 2024-09-13T11:56:48.112 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37809]: starting. 2024-09-13T11:56:48.112 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37809]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-37808 2024-09-13T11:56:48.112 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37809]: created object 0... 2024-09-13T11:56:48.112 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37809]: created object 25... 2024-09-13T11:56:48.112 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37809]: created object 49... 2024-09-13T11:56:48.112 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37809]: finishing. 2024-09-13T11:56:48.112 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37809]: shutting down. 2024-09-13T11:56:49.041 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37810]: starting. 2024-09-13T11:56:49.041 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37810]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-37808 2024-09-13T11:56:49.041 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37810]: shutting down. 2024-09-13T11:56:50.313 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:56:50.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.420+0000 7f03daabf640 10 monclient: get_monmap_and_config 2024-09-13T11:56:50.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.420+0000 7f03daabf640 10 monclient: build_initial_monmap 2024-09-13T11:56:50.421 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.420+0000 7f03daabf640 10 monclient: monmap: 2024-09-13T11:56:50.421 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:56:50.421 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:56:50.422 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:56:50.420989+0000 2024-09-13T11:56:50.422 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:56:50.420989+0000 2024-09-13T11:56:50.422 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:56:50.422 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:56:50.422 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:56:50.422 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:56:50.422 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:56:50.422 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:56:50.422 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:56:50.422 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:56:50.430 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.420+0000 7f03daabf640 1 Processor -- start 2024-09-13T11:56:50.430 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03daabf640 1 -- start start 2024-09-13T11:56:50.430 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03daabf640 10 monclient: init 2024-09-13T11:56:50.430 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03daabf640 10 monclient: _reopen_session rank -1 2024-09-13T11:56:50.430 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03daabf640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-13T11:56:50.430 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03daabf640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f03d406ea50 0x7f03d40a2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:56:50.430 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03daabf640 10 monclient(hunting): picked mon.noname-c con 0x7f03d406ea50 addr v2:172.21.15.122:3300/0 2024-09-13T11:56:50.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03daabf640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f03d406ee30 0x7f03d406f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:56:50.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03daabf640 10 monclient(hunting): picked mon.noname-a con 0x7f03d406ee30 addr v2:172.21.15.100:3300/0 2024-09-13T11:56:50.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03daabf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f03d406f7c0 addr v1:172.21.15.100:6789/0 2024-09-13T11:56:50.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03daabf640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03d40a1590 con 0x7f03d406f7c0 2024-09-13T11:56:50.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03daabf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:56:50.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03daabf640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f03d40a1870 con 0x7f03d406ee30 2024-09-13T11:56:50.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03daabf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:56:50.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03daabf640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f03d40a1b50 con 0x7f03d406ea50 2024-09-13T11:56:50.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03daabf640 10 monclient(hunting): _renew_subs 2024-09-13T11:56:50.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03daabf640 10 monclient(hunting): authenticate will time out at 991.014555s 2024-09-13T11:56:50.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d3fff640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f03d406ee30 0x7f03d406f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:56:50.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d3fff640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f03d406ee30 0x7f03d406f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:48272/0 (socket says 172.21.15.100:48272) 2024-09-13T11:56:50.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d3fff640 1 -- 172.21.15.100:0/3494331229 learned_addr learned my addr 172.21.15.100:0/3494331229 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:56:50.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d3fff640 10 monclient(hunting): get_auth_request con 0x7f03d406ee30 auth_method 0 2024-09-13T11:56:50.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:56:50.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:56:50.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:56:50.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:56:50.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:56:50.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d885b640 1 --2- 172.21.15.100:0/3494331229 >> v2:172.21.15.122:3300/0 conn(0x7f03d406ea50 0x7f03d40a2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:56:50.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:56:50.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d885b640 10 monclient(hunting): get_auth_request con 0x7f03d406ea50 auth_method 0 2024-09-13T11:56:50.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d885b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:56:50.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d885b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:56:50.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d885b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:56:50.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d885b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:56:50.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d885b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:56:50.433 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d885b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:56:50.433 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d3fff640 10 monclient(hunting): handle_auth_done global_id 14763 payload 274 2024-09-13T11:56:50.433 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d3fff640 1 -- 172.21.15.100:0/3494331229 >> v2:172.21.15.122:3300/0 conn(0x7f03d406ea50 msgr2=0x7f03d40a2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:56:50.433 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d3fff640 1 --2- 172.21.15.100:0/3494331229 >> v2:172.21.15.122:3300/0 conn(0x7f03d406ea50 0x7f03d40a2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:50.433 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d3fff640 1 -- 172.21.15.100:0/3494331229 >> v1:172.21.15.100:6789/0 conn(0x7f03d406f7c0 legacy=0x7f03d41375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:56:50.433 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d3fff640 10 monclient: _finish_hunting 0 2024-09-13T11:56:50.433 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d3fff640 1 monclient: found mon.noname-a 2024-09-13T11:56:50.433 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.421+0000 7f03d37fe640 1 -- 172.21.15.100:0/3494331229 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2918838039 0 0) 0x7f03d40a1590 con 0x7f03d406f7c0 2024-09-13T11:56:50.433 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.422+0000 7f03d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:56:50.433 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.422+0000 7f03d3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:56:50.433 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.422+0000 7f03d3fff640 1 -- 172.21.15.100:0/3494331229 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03d406fca0 con 0x7f03d406ee30 2024-09-13T11:56:50.433 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.422+0000 7f03d37fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:56:50.433 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.422+0000 7f03d3fff640 1 --2- 172.21.15.100:0/3494331229 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03d406ee30 0x7f03d406f280 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f03d40a7960 tx=0x7f03c402a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c22c225cb66a408 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:56:50.433 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.422+0000 7f03d37fe640 1 -- 172.21.15.100:0/3494331229 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c4037070 con 0x7f03d406ee30 2024-09-13T11:56:50.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.422+0000 7f03d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:56:50.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.422+0000 7f03d37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:56:50.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.422+0000 7f03d37fe640 10 monclient: dump: 2024-09-13T11:56:50.434 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:56:50.434 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:56:50.434 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:50.434 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:50.434 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:56:50.434 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:56:50.434 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:56:50.434 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:56:50.434 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:56:50.434 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:56:50.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.422+0000 7f03d37fe640 10 monclient: _finish_auth 0 2024-09-13T11:56:50.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.422+0000 7f03d37fe640 10 monclient: _check_auth_tickets 2024-09-13T11:56:50.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.422+0000 7f03d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:56:50.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.422+0000 7f03d37fe640 1 -- 172.21.15.100:0/3494331229 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03c4032440 con 0x7f03d406ee30 2024-09-13T11:56:50.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.422+0000 7f03daabf640 5 monclient: authenticate success, global_id 14763 2024-09-13T11:56:50.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.422+0000 7f03daabf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:56:50.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.422+0000 7f03d37fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:56:50.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.422+0000 7f03d37fe640 1 -- 172.21.15.100:0/3494331229 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c4032810 con 0x7f03d406ee30 2024-09-13T11:56:50.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.422+0000 7f03daabf640 10 monclient: get_monmap_and_config success 2024-09-13T11:56:50.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.422+0000 7f03daabf640 4 set_mon_vals no callback set 2024-09-13T11:56:50.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.422+0000 7f03d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:56:50.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.422+0000 7f03d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:56:50.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.422+0000 7f03d37fe640 10 monclient: dump: 2024-09-13T11:56:50.435 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:56:50.436 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:56:50.436 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:50.436 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:50.436 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:56:50.436 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:56:50.436 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:56:50.436 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:56:50.436 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:56:50.436 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:56:50.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.423+0000 7f03daabf640 10 monclient: shutdown 2024-09-13T11:56:50.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.423+0000 7f03daabf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:56:50.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.423+0000 7f03daabf640 1 -- 172.21.15.100:0/3494331229 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03d406ee30 msgr2=0x7f03d406f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:56:50.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.423+0000 7f03daabf640 1 --2- 172.21.15.100:0/3494331229 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03d406ee30 0x7f03d406f280 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f03d40a7960 tx=0x7f03c402a150 comp rx=0 tx=0).stop 2024-09-13T11:56:50.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.423+0000 7f03daabf640 1 -- 172.21.15.100:0/3494331229 shutdown_connections 2024-09-13T11:56:50.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.423+0000 7f03daabf640 1 --2- 172.21.15.100:0/3494331229 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03d406ee30 0x7f03d406f280 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:50.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.423+0000 7f03daabf640 1 --2- 172.21.15.100:0/3494331229 >> v2:172.21.15.122:3300/0 conn(0x7f03d406ea50 0x7f03d40a2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:50.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.423+0000 7f03daabf640 1 -- 172.21.15.100:0/3494331229 >> 172.21.15.100:0/3494331229 conn(0x7f03d40a68e0 msgr2=0x7f03d40a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:56:50.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.423+0000 7f03daabf640 1 -- 172.21.15.100:0/3494331229 shutdown_connections 2024-09-13T11:56:50.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.423+0000 7f03daabf640 1 -- 172.21.15.100:0/3494331229 wait complete. 2024-09-13T11:56:50.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.424+0000 7f03daabf640 10 monclient: build_initial_monmap 2024-09-13T11:56:50.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.424+0000 7f03daabf640 10 monclient: monmap: 2024-09-13T11:56:50.437 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:56:50.437 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:56:50.437 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:56:50.437 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:56:50.437 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:56:50.437 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:56:50.438 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:56:50.438 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:56:50.438 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:56:50.438 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:56:50.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.424+0000 7f03daabf640 1 Processor -- start 2024-09-13T11:56:50.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.424+0000 7f03daabf640 1 -- start start 2024-09-13T11:56:50.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.424+0000 7f03daabf640 10 monclient: init 2024-09-13T11:56:50.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03daabf640 10 monclient: _reopen_session rank -1 2024-09-13T11:56:50.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03daabf640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:56:50.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03daabf640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03d406ea50 0x7f03d41cca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:56:50.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03daabf640 10 monclient(hunting): picked mon.noname-a con 0x7f03d406ea50 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:56:50.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03daabf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03d406ee30 0x7f03d41ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:56:50.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03daabf640 10 monclient(hunting): picked mon.noname-b con 0x7f03d406ee30 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:56:50.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03daabf640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03d406f7c0 0x7f03d4123e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:56:50.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03daabf640 10 monclient(hunting): picked mon.noname-c con 0x7f03d406f7c0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:56:50.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03daabf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:56:50.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03daabf640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f03d4137e30 con 0x7f03d406ea50 2024-09-13T11:56:50.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03daabf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:56:50.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03daabf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f03d4137cb0 con 0x7f03d406ee30 2024-09-13T11:56:50.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03daabf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:56:50.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03daabf640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f03d4137fb0 con 0x7f03d406f7c0 2024-09-13T11:56:50.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03daabf640 10 monclient(hunting): _renew_subs 2024-09-13T11:56:50.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03daabf640 10 monclient(hunting): authenticate will time out at 991.018260s 2024-09-13T11:56:50.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d885b640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03d406ea50 0x7f03d41cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:56:50.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d885b640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03d406ea50 0x7f03d41cca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:48284/0 (socket says 172.21.15.100:48284) 2024-09-13T11:56:50.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d885b640 1 -- 172.21.15.100:0/3984600459 learned_addr learned my addr 172.21.15.100:0/3984600459 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:56:50.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d885b640 10 monclient(hunting): get_auth_request con 0x7f03d406ea50 auth_method 0 2024-09-13T11:56:50.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d885b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:56:50.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d885b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:56:50.440 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d885b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:56:50.440 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d905c640 1 --2- 172.21.15.100:0/3984600459 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03d406f7c0 0x7f03d4123e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:56:50.440 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d885b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:56:50.440 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d885b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:56:50.440 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d885b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:56:50.440 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d905c640 10 monclient(hunting): get_auth_request con 0x7f03d406f7c0 auth_method 0 2024-09-13T11:56:50.440 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d905c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:56:50.440 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d905c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:56:50.440 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d905c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:56:50.440 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d905c640 1 -- 172.21.15.100:0/3984600459 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03d406f7c0 msgr2=0x7f03d4123e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-13T11:56:50.440 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d905c640 1 -- 172.21.15.100:0/3984600459 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03d406f7c0 msgr2=0x7f03d4123e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T11:56:50.440 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d905c640 1 --2- 172.21.15.100:0/3984600459 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03d406f7c0 0x7f03d4123e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:56:50.441 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d905c640 1 --2- 172.21.15.100:0/3984600459 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03d406f7c0 0x7f03d4123e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:56:50.441 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d3fff640 1 --2- 172.21.15.100:0/3984600459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03d406ee30 0x7f03d41ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:56:50.441 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d885b640 10 monclient(hunting): handle_auth_done global_id 14769 payload 824 2024-09-13T11:56:50.441 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d885b640 1 -- 172.21.15.100:0/3984600459 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03d406f7c0 msgr2=0x7f03d4123e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:56:50.441 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d885b640 1 --2- 172.21.15.100:0/3984600459 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03d406f7c0 0x7f03d4123e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:50.441 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d885b640 1 -- 172.21.15.100:0/3984600459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03d406ee30 msgr2=0x7f03d41ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:56:50.441 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d885b640 1 --2- 172.21.15.100:0/3984600459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03d406ee30 0x7f03d41ccf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:50.441 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d885b640 10 monclient: _finish_hunting 0 2024-09-13T11:56:50.441 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d885b640 1 monclient: found mon.noname-a 2024-09-13T11:56:50.441 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d885b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:56:50.441 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d885b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:56:50.441 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d885b640 1 -- 172.21.15.100:0/3984600459 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03d41279a0 con 0x7f03d406ea50 2024-09-13T11:56:50.441 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d885b640 1 --2- 172.21.15.100:0/3984600459 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03d406ea50 0x7f03d41cca40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00e9b0 tx=0x7f03bc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:56:50.442 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d17fa640 1 -- 172.21.15.100:0/3984600459 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03bc017070 con 0x7f03d406ea50 2024-09-13T11:56:50.442 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:56:50.442 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:56:50.442 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d17fa640 10 monclient: dump: 2024-09-13T11:56:50.442 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:56:50.442 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:56:50.442 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:50.442 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:50.442 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:56:50.442 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:56:50.442 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:56:50.442 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:56:50.442 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:56:50.442 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:56:50.443 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d17fa640 10 monclient: _finish_auth 0 2024-09-13T11:56:50.443 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d17fa640 10 monclient: _check_auth_tickets 2024-09-13T11:56:50.443 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:56:50.443 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d17fa640 1 -- 172.21.15.100:0/3984600459 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03bc00c9a0 con 0x7f03d406ea50 2024-09-13T11:56:50.443 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03daabf640 5 monclient: authenticate success, global_id 14769 2024-09-13T11:56:50.443 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03daabf640 10 monclient: _renew_subs 2024-09-13T11:56:50.443 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03daabf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:56:50.443 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03daabf640 1 -- 172.21.15.100:0/3984600459 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03d4127c80 con 0x7f03d406ea50 2024-09-13T11:56:50.443 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d17fa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:56:50.443 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d17fa640 1 -- 172.21.15.100:0/3984600459 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03bc00cd90 con 0x7f03d406ea50 2024-09-13T11:56:50.443 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:56:50.443 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:56:50.443 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d17fa640 10 monclient: dump: 2024-09-13T11:56:50.444 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:56:50.444 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:56:50.444 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:50.444 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:50.444 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:56:50.444 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:56:50.444 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:56:50.444 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:56:50.444 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:56:50.444 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:56:50.444 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.425+0000 7f03d2ffd640 4 set_mon_vals no callback set 2024-09-13T11:56:50.444 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.426+0000 7f03daabf640 10 monclient: _renew_subs 2024-09-13T11:56:50.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.426+0000 7f03daabf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:56:50.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.426+0000 7f03daabf640 1 -- 172.21.15.100:0/3984600459 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03d41273e0 con 0x7f03d406ea50 2024-09-13T11:56:50.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.426+0000 7f03d17fa640 1 -- 172.21.15.100:0/3984600459 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f03bc01a070 con 0x7f03d406ea50 2024-09-13T11:56:50.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.426+0000 7f03daabf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:56:50.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.426+0000 7f03daabf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:56:50.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.426+0000 7f03daabf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:56:50.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.426+0000 7f03daabf640 1 -- 172.21.15.100:0/3984600459 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03d4138360 con 0x7f03d406ea50 2024-09-13T11:56:50.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.427+0000 7f03d17fa640 1 --2- 172.21.15.100:0/3984600459 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f03b403dfa0 0x7f03b4040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:56:50.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.427+0000 7f03d17fa640 1 -- 172.21.15.100:0/3984600459 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(209..209 src has 1..209) ==== 4738+0+0 (secure 0 0 0) 0x7f03bc052780 con 0x7f03d406ea50 2024-09-13T11:56:50.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.427+0000 7f03d3fff640 1 --2- 172.21.15.100:0/3984600459 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f03b403dfa0 0x7f03b4040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:56:50.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.427+0000 7f03d3fff640 10 monclient: get_auth_request con 0x7f03b403dfa0 auth_method 0 2024-09-13T11:56:50.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.427+0000 7f03d3fff640 1 --2- 172.21.15.100:0/3984600459 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f03b403dfa0 0x7f03b4040460 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f03c4034000 tx=0x7f03c402e000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:56:50.446 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.432+0000 7f03d17fa640 1 -- 172.21.15.100:0/3984600459 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f03bc054cf0 con 0x7f03d406ea50 2024-09-13T11:56:50.446 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.432+0000 7f03d17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:56:50.446 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.432+0000 7f03d17fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:56:50.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.671+0000 7f03daabf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:56:50.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.671+0000 7f03daabf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:56:50.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.671+0000 7f03daabf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:56:50.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.671+0000 7f03daabf640 1 -- 172.21.15.100:0/3984600459 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f03d41270b0 con 0x7f03d406ea50 2024-09-13T11:56:50.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.672+0000 7f03d17fa640 1 -- 172.21.15.100:0/3984600459 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f03bc013cc0 con 0x7f03d406ea50 2024-09-13T11:56:50.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.672+0000 7f03d17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:56:50.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.672+0000 7f03d17fa640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:56:50.674 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":20,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":94,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:56:50.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.674+0000 7f03daabf640 1 -- 172.21.15.100:0/3984600459 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f03b403dfa0 msgr2=0x7f03b4040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:56:50.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.674+0000 7f03daabf640 1 --2- 172.21.15.100:0/3984600459 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f03b403dfa0 0x7f03b4040460 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f03c4034000 tx=0x7f03c402e000 comp rx=0 tx=0).stop 2024-09-13T11:56:50.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.674+0000 7f03daabf640 10 monclient: shutdown 2024-09-13T11:56:50.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.674+0000 7f03daabf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:56:50.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.674+0000 7f03daabf640 1 -- 172.21.15.100:0/3984600459 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03d406ea50 msgr2=0x7f03d41cca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:56:50.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.674+0000 7f03daabf640 1 --2- 172.21.15.100:0/3984600459 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03d406ea50 0x7f03d41cca40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00e9b0 tx=0x7f03bc00ee70 comp rx=0 tx=0).stop 2024-09-13T11:56:50.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.674+0000 7f03daabf640 1 -- 172.21.15.100:0/3984600459 shutdown_connections 2024-09-13T11:56:50.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.674+0000 7f03daabf640 1 --2- 172.21.15.100:0/3984600459 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f03b403dfa0 0x7f03b4040460 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:50.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.674+0000 7f03daabf640 1 --2- 172.21.15.100:0/3984600459 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03d406f7c0 0x7f03d4123e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:50.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.674+0000 7f03daabf640 1 --2- 172.21.15.100:0/3984600459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03d406ee30 0x7f03d41ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:50.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.674+0000 7f03daabf640 1 --2- 172.21.15.100:0/3984600459 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03d406ea50 0x7f03d41cca40 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:50.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.674+0000 7f03daabf640 1 -- 172.21.15.100:0/3984600459 >> 172.21.15.100:0/3984600459 conn(0x7f03d40a68e0 msgr2=0x7f03d409da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:56:50.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.674+0000 7f03daabf640 1 -- 172.21.15.100:0/3984600459 shutdown_connections 2024-09-13T11:56:50.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:50.674+0000 7f03daabf640 1 -- 172.21.15.100:0/3984600459 wait complete. 2024-09-13T11:56:51.129 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:51.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37843]: starting. 2024-09-13T11:56:51.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37843]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-37808 2024-09-13T11:56:51.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37843]: created object 0... 2024-09-13T11:56:51.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37843]: created object 25... 2024-09-13T11:56:51.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37843]: created object 49... 2024-09-13T11:56:51.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37843]: finishing. 2024-09-13T11:56:51.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37843]: shutting down. 2024-09-13T11:56:51.141 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:51.141 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37845]: starting. 2024-09-13T11:56:51.141 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37845]: listing objects. 2024-09-13T11:56:51.141 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37845]: listed object 0... 2024-09-13T11:56:51.142 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37845]: listed object 25... 2024-09-13T11:56:51.142 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37845]: saw 50 objects 2024-09-13T11:56:51.142 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37845]: shutting down. 2024-09-13T11:56:52.059 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:52.059 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37844]: starting. 2024-09-13T11:56:52.059 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37844]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-37808 2024-09-13T11:56:52.059 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37844]: shutting down. 2024-09-13T11:56:52.060 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:52.060 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:52.060 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:56:52.063 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:56:53.686 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:56:53.794 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.793+0000 7fc4e35d0640 10 monclient: get_monmap_and_config 2024-09-13T11:56:53.794 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.793+0000 7fc4e35d0640 10 monclient: build_initial_monmap 2024-09-13T11:56:53.794 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.793+0000 7fc4e35d0640 10 monclient: monmap: 2024-09-13T11:56:53.794 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:56:53.794 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:56:53.794 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:56:53.793879+0000 2024-09-13T11:56:53.794 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:56:53.793879+0000 2024-09-13T11:56:53.794 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:56:53.794 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:56:53.795 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:56:53.795 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:56:53.795 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:56:53.795 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:56:53.795 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:56:53.795 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:56:53.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.793+0000 7fc4e35d0640 1 Processor -- start 2024-09-13T11:56:53.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.793+0000 7fc4e35d0640 1 -- start start 2024-09-13T11:56:53.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.793+0000 7fc4e35d0640 10 monclient: init 2024-09-13T11:56:53.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e35d0640 10 monclient: _reopen_session rank -1 2024-09-13T11:56:53.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e35d0640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-13T11:56:53.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e35d0640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc4dc0a4870 0x7fc4dc0a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:56:53.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e35d0640 10 monclient(hunting): picked mon.noname-a con 0x7fc4dc0a4870 addr v2:172.21.15.100:3300/0 2024-09-13T11:56:53.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e35d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc4dc0a5190 addr v1:172.21.15.122:6789/0 2024-09-13T11:56:53.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e35d0640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4dc137d40 0x7fc4dc13faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:56:53.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e35d0640 10 monclient(hunting): picked mon.noname-b con 0x7fc4dc137d40 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:56:53.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e35d0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4dc0a1df0 con 0x7fc4dc0a5190 2024-09-13T11:56:53.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e35d0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:56:53.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e35d0640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7fc4dc0a5960 con 0x7fc4dc0a4870 2024-09-13T11:56:53.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e35d0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:56:53.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e35d0640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fc4dc0a0ed0 con 0x7fc4dc137d40 2024-09-13T11:56:53.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e35d0640 10 monclient(hunting): _renew_subs 2024-09-13T11:56:53.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e35d0640 10 monclient(hunting): authenticate will time out at 994.387477s 2024-09-13T11:56:53.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e136c640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc4dc0a4870 0x7fc4dc0a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:56:53.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e1b6d640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4dc137d40 0x7fc4dc13faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:56:53.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e1b6d640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4dc137d40 0x7fc4dc13faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:59014/0 (socket says 172.21.15.100:59014) 2024-09-13T11:56:53.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e1b6d640 1 -- 172.21.15.100:0/3155851558 learned_addr learned my addr 172.21.15.100:0/3155851558 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:56:53.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e136c640 10 monclient(hunting): get_auth_request con 0x7fc4dc0a4870 auth_method 0 2024-09-13T11:56:53.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e136c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:56:53.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e136c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:56:53.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e136c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:56:53.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e1b6d640 10 monclient(hunting): get_auth_request con 0x7fc4dc137d40 auth_method 0 2024-09-13T11:56:53.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e1b6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:56:53.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e1b6d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:56:53.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e1b6d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:56:53.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e136c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:56:53.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e136c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:56:53.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e1b6d640 1 -- 172.21.15.100:0/3155851558 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4dc137d40 msgr2=0x7fc4dc13faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-13T11:56:53.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e1b6d640 1 -- 172.21.15.100:0/3155851558 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4dc137d40 msgr2=0x7fc4dc13faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:56:53.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e1b6d640 1 --2- 172.21.15.100:0/3155851558 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4dc137d40 0x7fc4dc13faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:56:53.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e136c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:56:53.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e1b6d640 1 --2- 172.21.15.100:0/3155851558 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4dc137d40 0x7fc4dc13faa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:56:53.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e136c640 10 monclient(hunting): handle_auth_done global_id 14784 payload 274 2024-09-13T11:56:53.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e136c640 1 -- 172.21.15.100:0/3155851558 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4dc137d40 msgr2=0x7fc4dc13faa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:56:53.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e136c640 1 --2- 172.21.15.100:0/3155851558 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4dc137d40 0x7fc4dc13faa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:53.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e136c640 1 -- 172.21.15.100:0/3155851558 >> v1:172.21.15.122:6789/0 conn(0x7fc4dc0a5190 legacy=0x7fc4dc137630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:56:53.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e136c640 10 monclient: _finish_hunting 0 2024-09-13T11:56:53.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e136c640 1 monclient: found mon.noname-a 2024-09-13T11:56:53.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e136c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:56:53.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e136c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:56:53.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.794+0000 7fc4e136c640 1 -- 172.21.15.100:0/3155851558 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4dc070410 con 0x7fc4dc0a4870 2024-09-13T11:56:53.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.795+0000 7fc4e136c640 1 --2- 172.21.15.100:0/3155851558 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4dc0a4870 0x7fc4dc0a4c50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc4d80097a0 tx=0x7fc4d802a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0876720f0ecc6de server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:56:53.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.795+0000 7fc4d3fff640 1 -- 172.21.15.100:0/3155851558 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8037070 con 0x7fc4dc0a4870 2024-09-13T11:56:53.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.795+0000 7fc4d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:56:53.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.795+0000 7fc4d3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:56:53.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.795+0000 7fc4d3fff640 10 monclient: dump: 2024-09-13T11:56:53.808 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:56:53.808 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:56:53.808 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:53.808 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:53.808 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:56:53.808 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:56:53.808 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:56:53.808 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:56:53.808 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:56:53.808 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:56:53.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.795+0000 7fc4d3fff640 10 monclient: _finish_auth 0 2024-09-13T11:56:53.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.795+0000 7fc4d3fff640 10 monclient: _check_auth_tickets 2024-09-13T11:56:53.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.795+0000 7fc4d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:56:53.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.795+0000 7fc4d3fff640 1 -- 172.21.15.100:0/3155851558 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4d802adb0 con 0x7fc4dc0a4870 2024-09-13T11:56:53.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.795+0000 7fc4e35d0640 5 monclient: authenticate success, global_id 14784 2024-09-13T11:56:53.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.795+0000 7fc4d3fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:56:53.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.795+0000 7fc4e35d0640 10 monclient: get_monmap_and_config success 2024-09-13T11:56:53.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.795+0000 7fc4d3fff640 1 -- 172.21.15.100:0/3155851558 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8032660 con 0x7fc4dc0a4870 2024-09-13T11:56:53.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.795+0000 7fc4e35d0640 4 set_mon_vals no callback set 2024-09-13T11:56:53.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.795+0000 7fc4d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:56:53.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.795+0000 7fc4d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:56:53.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.795+0000 7fc4d3fff640 10 monclient: dump: 2024-09-13T11:56:53.809 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:56:53.809 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:56:53.809 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:53.809 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:53.810 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:56:53.810 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:56:53.810 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:56:53.810 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:56:53.810 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:56:53.810 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:56:53.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.796+0000 7fc4e35d0640 10 monclient: shutdown 2024-09-13T11:56:53.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.796+0000 7fc4e35d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:56:53.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.796+0000 7fc4e35d0640 1 -- 172.21.15.100:0/3155851558 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4dc0a4870 msgr2=0x7fc4dc0a4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:56:53.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.796+0000 7fc4e35d0640 1 --2- 172.21.15.100:0/3155851558 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4dc0a4870 0x7fc4dc0a4c50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc4d80097a0 tx=0x7fc4d802a060 comp rx=0 tx=0).stop 2024-09-13T11:56:53.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.796+0000 7fc4e35d0640 1 -- 172.21.15.100:0/3155851558 shutdown_connections 2024-09-13T11:56:53.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.796+0000 7fc4e35d0640 1 --2- 172.21.15.100:0/3155851558 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4dc137d40 0x7fc4dc13faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:53.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.796+0000 7fc4e35d0640 1 --2- 172.21.15.100:0/3155851558 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4dc0a4870 0x7fc4dc0a4c50 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:53.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.796+0000 7fc4e35d0640 1 -- 172.21.15.100:0/3155851558 >> 172.21.15.100:0/3155851558 conn(0x7fc4dc0a6970 msgr2=0x7fc4dc0a6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:56:53.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.796+0000 7fc4e35d0640 1 -- 172.21.15.100:0/3155851558 shutdown_connections 2024-09-13T11:56:53.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.796+0000 7fc4e35d0640 1 -- 172.21.15.100:0/3155851558 wait complete. 2024-09-13T11:56:53.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.797+0000 7fc4e35d0640 10 monclient: build_initial_monmap 2024-09-13T11:56:53.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.797+0000 7fc4e35d0640 10 monclient: monmap: 2024-09-13T11:56:53.811 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:56:53.811 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:56:53.811 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:56:53.811 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:56:53.811 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:56:53.811 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:56:53.811 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:56:53.811 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:56:53.811 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:56:53.812 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:56:53.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.797+0000 7fc4e35d0640 1 Processor -- start 2024-09-13T11:56:53.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.797+0000 7fc4e35d0640 1 -- start start 2024-09-13T11:56:53.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.797+0000 7fc4e35d0640 10 monclient: init 2024-09-13T11:56:53.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.797+0000 7fc4e35d0640 10 monclient: _reopen_session rank -1 2024-09-13T11:56:53.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e35d0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:56:53.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e35d0640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4dc0a4870 0x7fc4dc123e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:56:53.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e35d0640 10 monclient(hunting): picked mon.noname-a con 0x7fc4dc0a4870 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:56:53.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e35d0640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4dc0a5190 0x7fc4dc124350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:56:53.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e35d0640 10 monclient(hunting): picked mon.noname-c con 0x7fc4dc0a5190 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:56:53.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e35d0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4dc137d40 0x7fc4dc124890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:56:53.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e35d0640 10 monclient(hunting): picked mon.noname-b con 0x7fc4dc137d40 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:56:53.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e35d0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:56:53.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e35d0640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fc4dc06fc60 con 0x7fc4dc0a4870 2024-09-13T11:56:53.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e35d0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:56:53.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e35d0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc4dc0a55a0 con 0x7fc4dc137d40 2024-09-13T11:56:53.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e35d0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:56:53.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e35d0640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fc4dc06fde0 con 0x7fc4dc0a5190 2024-09-13T11:56:53.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e35d0640 10 monclient(hunting): _renew_subs 2024-09-13T11:56:53.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e35d0640 10 monclient(hunting): authenticate will time out at 994.391148s 2024-09-13T11:56:53.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e136c640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4dc0a4870 0x7fc4dc123e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:56:53.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e136c640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4dc0a4870 0x7fc4dc123e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47088/0 (socket says 172.21.15.100:47088) 2024-09-13T11:56:53.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e136c640 1 -- 172.21.15.100:0/201284526 learned_addr learned my addr 172.21.15.100:0/201284526 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:56:53.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e136c640 10 monclient(hunting): get_auth_request con 0x7fc4dc0a4870 auth_method 0 2024-09-13T11:56:53.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e136c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:56:53.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e136c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:56:53.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e136c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:56:53.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e0b6b640 1 --2- 172.21.15.100:0/201284526 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4dc0a5190 0x7fc4dc124350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:56:53.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e0b6b640 10 monclient(hunting): get_auth_request con 0x7fc4dc0a5190 auth_method 0 2024-09-13T11:56:53.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e0b6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:56:53.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e0b6b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:56:53.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e0b6b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:56:53.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e136c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:56:53.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e136c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:56:53.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e136c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:56:53.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e0b6b640 1 -- 172.21.15.100:0/201284526 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4dc0a5190 msgr2=0x7fc4dc124350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-13T11:56:53.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e0b6b640 1 -- 172.21.15.100:0/201284526 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4dc0a5190 msgr2=0x7fc4dc124350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T11:56:53.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e0b6b640 1 --2- 172.21.15.100:0/201284526 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4dc0a5190 0x7fc4dc124350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:56:53.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e0b6b640 1 --2- 172.21.15.100:0/201284526 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4dc0a5190 0x7fc4dc124350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:56:53.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e1b6d640 1 --2- 172.21.15.100:0/201284526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4dc137d40 0x7fc4dc124890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:56:53.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e1b6d640 10 monclient(hunting): get_auth_request con 0x7fc4dc137d40 auth_method 0 2024-09-13T11:56:53.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e1b6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:56:53.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e1b6d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:56:53.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e1b6d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:56:53.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e1b6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:56:53.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e1b6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:56:53.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e1b6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:56:53.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e136c640 10 monclient(hunting): handle_auth_done global_id 14787 payload 824 2024-09-13T11:56:53.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e136c640 1 -- 172.21.15.100:0/201284526 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4dc0a5190 msgr2=0x7fc4dc124350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:56:53.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e136c640 1 --2- 172.21.15.100:0/201284526 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4dc0a5190 0x7fc4dc124350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:53.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e136c640 1 -- 172.21.15.100:0/201284526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4dc137d40 msgr2=0x7fc4dc124890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:56:53.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e136c640 1 --2- 172.21.15.100:0/201284526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4dc137d40 0x7fc4dc124890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:53.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e136c640 10 monclient: _finish_hunting 0 2024-09-13T11:56:53.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e136c640 1 monclient: found mon.noname-a 2024-09-13T11:56:53.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e136c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:56:53.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e136c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:56:53.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e136c640 1 -- 172.21.15.100:0/201284526 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4dc127910 con 0x7fc4dc0a4870 2024-09-13T11:56:53.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e136c640 1 --2- 172.21.15.100:0/201284526 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4dc0a4870 0x7fc4dc123e10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8035040 tx=0x7fc4d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:56:53.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4d1ffb640 1 -- 172.21.15.100:0/201284526 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8041070 con 0x7fc4dc0a4870 2024-09-13T11:56:53.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:56:53.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4d1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:56:53.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4d1ffb640 10 monclient: dump: 2024-09-13T11:56:53.816 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:56:53.816 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:56:53.816 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:53.816 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:53.816 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:56:53.817 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:56:53.817 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:56:53.817 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:56:53.817 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:56:53.817 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:56:53.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4d1ffb640 10 monclient: _finish_auth 0 2024-09-13T11:56:53.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4d1ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:56:53.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:56:53.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4d1ffb640 1 -- 172.21.15.100:0/201284526 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4d803c670 con 0x7fc4dc0a4870 2024-09-13T11:56:53.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e35d0640 5 monclient: authenticate success, global_id 14787 2024-09-13T11:56:53.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4d1ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:56:53.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e35d0640 10 monclient: _renew_subs 2024-09-13T11:56:53.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4d1ffb640 1 -- 172.21.15.100:0/201284526 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8037040 con 0x7fc4dc0a4870 2024-09-13T11:56:53.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e35d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:56:53.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4e35d0640 1 -- 172.21.15.100:0/201284526 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4dc127bf0 con 0x7fc4dc0a4870 2024-09-13T11:56:53.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:56:53.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4d1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:56:53.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4d1ffb640 10 monclient: dump: 2024-09-13T11:56:53.818 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:56:53.818 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:56:53.818 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:53.818 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:53.818 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:56:53.818 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:56:53.818 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:56:53.818 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:56:53.818 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:56:53.819 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:56:53.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.798+0000 7fc4d37fe640 4 set_mon_vals no callback set 2024-09-13T11:56:53.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.799+0000 7fc4e35d0640 10 monclient: _renew_subs 2024-09-13T11:56:53.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.799+0000 7fc4e35d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:56:53.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.799+0000 7fc4e35d0640 1 -- 172.21.15.100:0/201284526 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4dc127630 con 0x7fc4dc0a4870 2024-09-13T11:56:53.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.800+0000 7fc4d1ffb640 1 -- 172.21.15.100:0/201284526 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fc4d803a070 con 0x7fc4dc0a4870 2024-09-13T11:56:53.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.800+0000 7fc4e35d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:56:53.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.800+0000 7fc4e35d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:56:53.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.800+0000 7fc4e35d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:56:53.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.800+0000 7fc4e35d0640 1 -- 172.21.15.100:0/201284526 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4dc0a28f0 con 0x7fc4dc0a4870 2024-09-13T11:56:53.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.800+0000 7fc4d1ffb640 1 --2- 172.21.15.100:0/201284526 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc4c003dfa0 0x7fc4c0040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:56:53.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.800+0000 7fc4d1ffb640 1 -- 172.21.15.100:0/201284526 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(212..212 src has 1..212) ==== 4738+0+0 (secure 0 0 0) 0x7fc4d8072b40 con 0x7fc4dc0a4870 2024-09-13T11:56:53.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.800+0000 7fc4e0b6b640 1 --2- 172.21.15.100:0/201284526 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc4c003dfa0 0x7fc4c0040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:56:53.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.800+0000 7fc4e0b6b640 10 monclient: get_auth_request con 0x7fc4c003dfa0 auth_method 0 2024-09-13T11:56:53.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.801+0000 7fc4e0b6b640 1 --2- 172.21.15.100:0/201284526 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc4c003dfa0 0x7fc4c0040460 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc125be0 tx=0x7fc4cc009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:56:53.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.805+0000 7fc4d1ffb640 1 -- 172.21.15.100:0/201284526 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc4d803ac60 con 0x7fc4dc0a4870 2024-09-13T11:56:53.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.805+0000 7fc4d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:56:53.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:53.805+0000 7fc4d1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:56:54.056 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:54.055+0000 7fc4e35d0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:56:54.056 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:54.055+0000 7fc4e35d0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:56:54.056 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:54.055+0000 7fc4e35d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:56:54.056 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:54.055+0000 7fc4e35d0640 1 -- 172.21.15.100:0/201284526 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc4dc0a10d0 con 0x7fc4dc0a4870 2024-09-13T11:56:54.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:54.056+0000 7fc4d1ffb640 1 -- 172.21.15.100:0/201284526 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fc4d8030d50 con 0x7fc4dc0a4870 2024-09-13T11:56:54.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:54.056+0000 7fc4d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:56:54.058 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":20,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":97,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:56:54.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:54.056+0000 7fc4d1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:56:54.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:54.059+0000 7fc4e35d0640 1 -- 172.21.15.100:0/201284526 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc4c003dfa0 msgr2=0x7fc4c0040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:56:54.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:54.059+0000 7fc4e35d0640 1 --2- 172.21.15.100:0/201284526 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc4c003dfa0 0x7fc4c0040460 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc125be0 tx=0x7fc4cc009040 comp rx=0 tx=0).stop 2024-09-13T11:56:54.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:54.059+0000 7fc4e35d0640 10 monclient: shutdown 2024-09-13T11:56:54.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:54.059+0000 7fc4e35d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:56:54.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:54.059+0000 7fc4e35d0640 1 -- 172.21.15.100:0/201284526 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4dc0a4870 msgr2=0x7fc4dc123e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:56:54.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:54.059+0000 7fc4e35d0640 1 --2- 172.21.15.100:0/201284526 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4dc0a4870 0x7fc4dc123e10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8035040 tx=0x7fc4d80046c0 comp rx=0 tx=0).stop 2024-09-13T11:56:54.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:54.059+0000 7fc4e35d0640 1 -- 172.21.15.100:0/201284526 shutdown_connections 2024-09-13T11:56:54.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:54.059+0000 7fc4e35d0640 1 --2- 172.21.15.100:0/201284526 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc4c003dfa0 0x7fc4c0040460 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:54.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:54.059+0000 7fc4e35d0640 1 --2- 172.21.15.100:0/201284526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4dc137d40 0x7fc4dc124890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:54.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:54.059+0000 7fc4e35d0640 1 --2- 172.21.15.100:0/201284526 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4dc0a5190 0x7fc4dc124350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:54.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:54.059+0000 7fc4e35d0640 1 --2- 172.21.15.100:0/201284526 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4dc0a4870 0x7fc4dc123e10 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:54.062 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:54.059+0000 7fc4e35d0640 1 -- 172.21.15.100:0/201284526 >> 172.21.15.100:0/201284526 conn(0x7fc4dc0a6970 msgr2=0x7fc4dc06f130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:56:54.062 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:54.059+0000 7fc4e35d0640 1 -- 172.21.15.100:0/201284526 shutdown_connections 2024-09-13T11:56:54.062 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:54.059+0000 7fc4e35d0640 1 -- 172.21.15.100:0/201284526 wait complete. 2024-09-13T11:56:54.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37912]: starting. 2024-09-13T11:56:54.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37912]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-37911 2024-09-13T11:56:54.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37912]: created object 0... 2024-09-13T11:56:54.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37912]: created object 25... 2024-09-13T11:56:54.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37912]: created object 49... 2024-09-13T11:56:54.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37912]: finishing. 2024-09-13T11:56:54.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[37912]: shutting down. 2024-09-13T11:56:55.037 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37913]: starting. 2024-09-13T11:56:55.037 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37913]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-37911 2024-09-13T11:56:55.037 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[37913]: shutting down. 2024-09-13T11:56:57.072 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:56:57.118 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:57.118 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37946]: starting. 2024-09-13T11:56:57.118 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37946]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-37911 2024-09-13T11:56:57.118 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37946]: created object 0... 2024-09-13T11:56:57.118 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37946]: created object 25... 2024-09-13T11:56:57.118 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37946]: created object 49... 2024-09-13T11:56:57.118 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37946]: finishing. 2024-09-13T11:56:57.118 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[37946]: shutting down. 2024-09-13T11:56:57.130 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:57.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37948]: starting. 2024-09-13T11:56:57.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37948]: listing objects. 2024-09-13T11:56:57.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37948]: listed object 0... 2024-09-13T11:56:57.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37948]: listed object 25... 2024-09-13T11:56:57.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37948]: saw 50 objects 2024-09-13T11:56:57.130 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[37948]: shutting down. 2024-09-13T11:56:57.178 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.177+0000 7f51946db640 10 monclient: get_monmap_and_config 2024-09-13T11:56:57.178 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.177+0000 7f51946db640 10 monclient: build_initial_monmap 2024-09-13T11:56:57.178 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.177+0000 7f51946db640 10 monclient: monmap: 2024-09-13T11:56:57.178 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:56:57.178 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:56:57.178 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:56:57.177940+0000 2024-09-13T11:56:57.178 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:56:57.177940+0000 2024-09-13T11:56:57.179 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:56:57.179 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:56:57.179 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:56:57.179 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:56:57.179 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:56:57.179 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:56:57.179 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:56:57.179 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:56:57.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.177+0000 7f51946db640 1 Processor -- start 2024-09-13T11:56:57.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.177+0000 7f51946db640 1 -- start start 2024-09-13T11:56:57.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.177+0000 7f51946db640 10 monclient: init 2024-09-13T11:56:57.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f51946db640 10 monclient: _reopen_session rank -1 2024-09-13T11:56:57.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f51946db640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-13T11:56:57.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f51946db640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f518c06ea50 0x7f518c0a2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:56:57.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f51946db640 10 monclient(hunting): picked mon.noname-b con 0x7f518c06ea50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:56:57.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f51946db640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f518c06ee30 0x7f518c06f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:56:57.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f51946db640 10 monclient(hunting): picked mon.noname-c con 0x7f518c06ee30 addr v2:172.21.15.122:3300/0 2024-09-13T11:56:57.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f51946db640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f518c06f790 addr v1:172.21.15.122:6789/0 2024-09-13T11:56:57.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f51946db640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f518c0a1590 con 0x7f518c06f790 2024-09-13T11:56:57.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f51946db640 10 monclient(hunting): start opening mon connection 2024-09-13T11:56:57.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f51946db640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f518c0a1870 con 0x7f518c06ee30 2024-09-13T11:56:57.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f51946db640 10 monclient(hunting): start opening mon connection 2024-09-13T11:56:57.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f51946db640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f518c0a1b50 con 0x7f518c06ea50 2024-09-13T11:56:57.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f51946db640 10 monclient(hunting): _renew_subs 2024-09-13T11:56:57.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f51946db640 10 monclient(hunting): authenticate will time out at 997.771499s 2024-09-13T11:56:57.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f5192477640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f518c06ea50 0x7f518c0a2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:56:57.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f5192477640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f518c06ea50 0x7f518c0a2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:59028/0 (socket says 172.21.15.100:59028) 2024-09-13T11:56:57.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f5192477640 1 -- 172.21.15.100:0/594261833 learned_addr learned my addr 172.21.15.100:0/594261833 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:56:57.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f5192477640 10 monclient(hunting): get_auth_request con 0x7f518c06ea50 auth_method 0 2024-09-13T11:56:57.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f5192477640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:56:57.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f5192477640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:56:57.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f5192477640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:56:57.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f5192477640 1 -- 172.21.15.100:0/594261833 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f518c06ea50 msgr2=0x7f518c0a2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:56:57.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f5192477640 1 -- 172.21.15.100:0/594261833 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f518c06ea50 msgr2=0x7f518c0a2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:56:57.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f5192477640 1 --2- 172.21.15.100:0/594261833 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f518c06ea50 0x7f518c0a2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:56:57.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f5192477640 1 --2- 172.21.15.100:0/594261833 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f518c06ea50 0x7f518c0a2860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:56:57.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f5191c76640 1 --2- 172.21.15.100:0/594261833 >> v2:172.21.15.122:3300/0 conn(0x7f518c06ee30 0x7f518c06f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:56:57.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f5191c76640 10 monclient(hunting): get_auth_request con 0x7f518c06ee30 auth_method 0 2024-09-13T11:56:57.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f5191c76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:56:57.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f5191c76640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:56:57.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f5191c76640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:56:57.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f5191c76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:56:57.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f5191c76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:56:57.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.178+0000 7f5191c76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:56:57.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.179+0000 7f5191c76640 10 monclient(hunting): handle_auth_done global_id 5590 payload 274 2024-09-13T11:56:57.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.179+0000 7f5191c76640 1 -- 172.21.15.100:0/594261833 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f518c06ea50 msgr2=0x7f518c0a2860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:56:57.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.179+0000 7f5191c76640 1 --2- 172.21.15.100:0/594261833 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f518c06ea50 0x7f518c0a2860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:57.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.179+0000 7f5191c76640 1 -- 172.21.15.100:0/594261833 >> v1:172.21.15.122:6789/0 conn(0x7f518c06f790 legacy=0x7f518c1375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:56:57.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.179+0000 7f5191c76640 10 monclient: _finish_hunting 0 2024-09-13T11:56:57.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.179+0000 7f5191c76640 1 monclient: found mon.noname-c 2024-09-13T11:56:57.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.179+0000 7f5191c76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:56:57.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.179+0000 7f5191475640 1 -- 172.21.15.100:0/594261833 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1965087795 0 0) 0x7f518c0a1590 con 0x7f518c06f790 2024-09-13T11:56:57.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.179+0000 7f5191c76640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.122:3300/0 2024-09-13T11:56:57.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.179+0000 7f5191c76640 1 -- 172.21.15.100:0/594261833 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f518c06fca0 con 0x7f518c06ee30 2024-09-13T11:56:57.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.179+0000 7f5191475640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:56:57.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.179+0000 7f5191c76640 1 --2- 172.21.15.100:0/594261833 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f518c06ee30 0x7f518c06f250 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f51800097a0 tx=0x7f518002a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfe081d47958aaa5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:56:57.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.179+0000 7f5191475640 1 -- 172.21.15.100:0/594261833 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5180037070 con 0x7f518c06ee30 2024-09-13T11:56:57.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.179+0000 7f5191475640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:56:57.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f5191475640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:56:57.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f5191475640 10 monclient: dump: 2024-09-13T11:56:57.192 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:56:57.192 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:56:57.193 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:57.193 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:57.193 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:56:57.193 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:56:57.193 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:56:57.193 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:56:57.193 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:56:57.193 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:56:57.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f5191475640 10 monclient: _finish_auth 0 2024-09-13T11:56:57.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f5191475640 10 monclient: _check_auth_tickets 2024-09-13T11:56:57.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f5191475640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:56:57.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f51946db640 5 monclient: authenticate success, global_id 5590 2024-09-13T11:56:57.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f51946db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:56:57.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f5191475640 1 -- 172.21.15.100:0/594261833 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f518002adb0 con 0x7f518c06ee30 2024-09-13T11:56:57.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f5191475640 10 monclient: handle_config config(0 keys) 2024-09-13T11:56:57.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f51946db640 10 monclient: get_monmap_and_config success 2024-09-13T11:56:57.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f51946db640 4 set_mon_vals no callback set 2024-09-13T11:56:57.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f5191475640 1 -- 172.21.15.100:0/594261833 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5180032660 con 0x7f518c06ee30 2024-09-13T11:56:57.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f5191475640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:56:57.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f5191475640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:56:57.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f5191475640 10 monclient: dump: 2024-09-13T11:56:57.194 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:56:57.194 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:56:57.194 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:57.194 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:57.194 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:56:57.194 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:56:57.195 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:56:57.195 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:56:57.195 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:56:57.195 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:56:57.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f51946db640 10 monclient: shutdown 2024-09-13T11:56:57.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f51946db640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:56:57.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f51946db640 1 -- 172.21.15.100:0/594261833 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f518c06ee30 msgr2=0x7f518c06f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:56:57.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f51946db640 1 --2- 172.21.15.100:0/594261833 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f518c06ee30 0x7f518c06f250 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f51800097a0 tx=0x7f518002a060 comp rx=0 tx=0).stop 2024-09-13T11:56:57.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f51946db640 1 -- 172.21.15.100:0/594261833 shutdown_connections 2024-09-13T11:56:57.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f51946db640 1 --2- 172.21.15.100:0/594261833 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f518c06ee30 0x7f518c06f250 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:57.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f51946db640 1 --2- 172.21.15.100:0/594261833 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f518c06ea50 0x7f518c0a2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:57.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f51946db640 1 -- 172.21.15.100:0/594261833 >> 172.21.15.100:0/594261833 conn(0x7f518c0a68e0 msgr2=0x7f518c0a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:56:57.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f51946db640 1 -- 172.21.15.100:0/594261833 shutdown_connections 2024-09-13T11:56:57.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.180+0000 7f51946db640 1 -- 172.21.15.100:0/594261833 wait complete. 2024-09-13T11:56:57.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.181+0000 7f51946db640 10 monclient: build_initial_monmap 2024-09-13T11:56:57.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.181+0000 7f51946db640 10 monclient: monmap: 2024-09-13T11:56:57.196 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:56:57.196 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:56:57.196 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:56:57.196 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:56:57.196 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:56:57.196 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:56:57.196 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:56:57.196 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:56:57.196 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:56:57.196 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:56:57.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.181+0000 7f51946db640 1 Processor -- start 2024-09-13T11:56:57.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.181+0000 7f51946db640 1 -- start start 2024-09-13T11:56:57.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.181+0000 7f51946db640 10 monclient: init 2024-09-13T11:56:57.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.181+0000 7f51946db640 10 monclient: _reopen_session rank -1 2024-09-13T11:56:57.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.181+0000 7f51946db640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:56:57.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f51946db640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f518c06ea50 0x7f518c123b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:56:57.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f51946db640 10 monclient(hunting): picked mon.noname-b con 0x7f518c06ea50 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:56:57.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f51946db640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f518c06ee30 0x7f518c1240a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:56:57.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f51946db640 10 monclient(hunting): picked mon.noname-a con 0x7f518c06ee30 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:56:57.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f51946db640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f518c06f790 0x7f518c1245e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:56:57.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f51946db640 10 monclient(hunting): picked mon.noname-c con 0x7f518c06f790 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:56:57.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f51946db640 10 monclient(hunting): start opening mon connection 2024-09-13T11:56:57.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f51946db640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f518c137e30 con 0x7f518c06ee30 2024-09-13T11:56:57.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f51946db640 10 monclient(hunting): start opening mon connection 2024-09-13T11:56:57.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f51946db640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f518c137cb0 con 0x7f518c06ea50 2024-09-13T11:56:57.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f51946db640 10 monclient(hunting): start opening mon connection 2024-09-13T11:56:57.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f51946db640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f518c137fb0 con 0x7f518c06f790 2024-09-13T11:56:57.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f51946db640 10 monclient(hunting): _renew_subs 2024-09-13T11:56:57.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f51946db640 10 monclient(hunting): authenticate will time out at 997.775062s 2024-09-13T11:56:57.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5191c76640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f518c06ee30 0x7f518c1240a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:56:57.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5192c78640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f518c06f790 0x7f518c1245e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:56:57.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5191c76640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f518c06ee30 0x7f518c1240a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47096/0 (socket says 172.21.15.100:47096) 2024-09-13T11:56:57.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5191c76640 1 -- 172.21.15.100:0/555219232 learned_addr learned my addr 172.21.15.100:0/555219232 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:56:57.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5191c76640 10 monclient(hunting): get_auth_request con 0x7f518c06ee30 auth_method 0 2024-09-13T11:56:57.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5191c76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:56:57.199 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5191c76640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:56:57.199 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5191c76640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:56:57.199 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5192c78640 10 monclient(hunting): get_auth_request con 0x7f518c06f790 auth_method 0 2024-09-13T11:56:57.199 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5192c78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:56:57.199 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5192c78640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:56:57.199 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5192c78640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:56:57.199 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5191c76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:56:57.199 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5191c76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:56:57.199 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5192c78640 1 -- 172.21.15.100:0/555219232 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f518c06f790 msgr2=0x7f518c1245e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-13T11:56:57.199 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5191c76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:56:57.199 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5192c78640 1 -- 172.21.15.100:0/555219232 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f518c06f790 msgr2=0x7f518c1245e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T11:56:57.199 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5192c78640 1 --2- 172.21.15.100:0/555219232 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f518c06f790 0x7f518c1245e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:56:57.199 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5192477640 1 --2- 172.21.15.100:0/555219232 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f518c06ea50 0x7f518c123b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:56:57.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5192c78640 1 --2- 172.21.15.100:0/555219232 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f518c06f790 0x7f518c1245e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:56:57.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5192477640 10 monclient(hunting): get_auth_request con 0x7f518c06ea50 auth_method 0 2024-09-13T11:56:57.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5192477640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:56:57.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5192477640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:56:57.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5192477640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:56:57.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5192477640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:56:57.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5192477640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:56:57.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5192477640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:56:57.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5191c76640 10 monclient(hunting): handle_auth_done global_id 14808 payload 824 2024-09-13T11:56:57.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5191c76640 1 -- 172.21.15.100:0/555219232 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f518c06f790 msgr2=0x7f518c1245e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:56:57.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5191c76640 1 --2- 172.21.15.100:0/555219232 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f518c06f790 0x7f518c1245e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:57.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5191c76640 1 -- 172.21.15.100:0/555219232 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f518c06ea50 msgr2=0x7f518c123b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:56:57.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5191c76640 1 --2- 172.21.15.100:0/555219232 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f518c06ea50 0x7f518c123b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:57.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5191c76640 10 monclient: _finish_hunting 0 2024-09-13T11:56:57.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5191c76640 1 monclient: found mon.noname-a 2024-09-13T11:56:57.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5191c76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:56:57.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5191c76640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:56:57.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5191c76640 1 -- 172.21.15.100:0/555219232 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f518c126b10 con 0x7f518c06ee30 2024-09-13T11:56:57.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5191c76640 1 --2- 172.21.15.100:0/555219232 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f518c06ee30 0x7f518c1240a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5180000c00 tx=0x7f51800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:56:57.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f518affd640 1 -- 172.21.15.100:0/555219232 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5180041070 con 0x7f518c06ee30 2024-09-13T11:56:57.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f518affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:56:57.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f518affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:56:57.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f518affd640 10 monclient: dump: 2024-09-13T11:56:57.201 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:56:57.201 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:56:57.201 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:57.202 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:57.202 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:56:57.202 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:56:57.202 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:56:57.202 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:56:57.202 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:56:57.202 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:56:57.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f518affd640 10 monclient: _finish_auth 0 2024-09-13T11:56:57.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f518affd640 10 monclient: _check_auth_tickets 2024-09-13T11:56:57.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f518affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:56:57.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f51946db640 5 monclient: authenticate success, global_id 14808 2024-09-13T11:56:57.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f518affd640 1 -- 172.21.15.100:0/555219232 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f518003c670 con 0x7f518c06ee30 2024-09-13T11:56:57.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f51946db640 10 monclient: _renew_subs 2024-09-13T11:56:57.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f51946db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:56:57.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f51946db640 1 -- 172.21.15.100:0/555219232 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f518c127980 con 0x7f518c06ee30 2024-09-13T11:56:57.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f518affd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:56:57.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f518affd640 1 -- 172.21.15.100:0/555219232 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5180037040 con 0x7f518c06ee30 2024-09-13T11:56:57.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f518affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:56:57.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f518affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:56:57.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f518affd640 10 monclient: dump: 2024-09-13T11:56:57.203 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:56:57.203 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:56:57.203 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:57.203 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:56:57.203 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:56:57.203 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:56:57.203 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:56:57.203 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:56:57.204 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:56:57.204 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:56:57.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f51946db640 10 monclient: _renew_subs 2024-09-13T11:56:57.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f5190c74640 4 set_mon_vals no callback set 2024-09-13T11:56:57.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f51946db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:56:57.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.182+0000 7f51946db640 1 -- 172.21.15.100:0/555219232 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f518c125620 con 0x7f518c06ee30 2024-09-13T11:56:57.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.183+0000 7f51946db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:56:57.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.183+0000 7f51946db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:56:57.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.183+0000 7f51946db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:56:57.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.183+0000 7f51946db640 1 -- 172.21.15.100:0/555219232 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f518c0a2860 con 0x7f518c06ee30 2024-09-13T11:56:57.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.184+0000 7f518affd640 1 -- 172.21.15.100:0/555219232 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f5180044070 con 0x7f518c06ee30 2024-09-13T11:56:57.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.184+0000 7f518affd640 1 --2- 172.21.15.100:0/555219232 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f517803dfa0 0x7f5178040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:56:57.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.184+0000 7f518affd640 1 -- 172.21.15.100:0/555219232 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(216..216 src has 1..216) ==== 4738+0+0 (secure 0 0 0) 0x7f5180072fa0 con 0x7f518c06ee30 2024-09-13T11:56:57.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.184+0000 7f5192477640 1 --2- 172.21.15.100:0/555219232 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f517803dfa0 0x7f5178040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:56:57.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.184+0000 7f5192477640 10 monclient: get_auth_request con 0x7f517803dfa0 auth_method 0 2024-09-13T11:56:57.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.184+0000 7f5192477640 1 --2- 172.21.15.100:0/555219232 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f517803dfa0 0x7f5178040460 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5184009100 tx=0x7f5184008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:56:57.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.189+0000 7f518affd640 1 -- 172.21.15.100:0/555219232 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f518003d3f0 con 0x7f518c06ee30 2024-09-13T11:56:57.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.189+0000 7f518affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:56:57.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.189+0000 7f518affd640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:56:57.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.430+0000 7f51946db640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:56:57.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.430+0000 7f51946db640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:56:57.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.430+0000 7f51946db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:56:57.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.430+0000 7f51946db640 1 -- 172.21.15.100:0/555219232 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f518c069100 con 0x7f518c06ee30 2024-09-13T11:56:57.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.430+0000 7f518affd640 1 -- 172.21.15.100:0/555219232 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7f518003ac80 con 0x7f518c06ee30 2024-09-13T11:56:57.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.430+0000 7f518affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:56:57.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.430+0000 7f518affd640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:56:57.432 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":20,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":101,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:56:57.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.432+0000 7f51946db640 1 -- 172.21.15.100:0/555219232 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f517803dfa0 msgr2=0x7f5178040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:56:57.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.432+0000 7f51946db640 1 --2- 172.21.15.100:0/555219232 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f517803dfa0 0x7f5178040460 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5184009100 tx=0x7f5184008040 comp rx=0 tx=0).stop 2024-09-13T11:56:57.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.432+0000 7f51946db640 10 monclient: shutdown 2024-09-13T11:56:57.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.432+0000 7f51946db640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:56:57.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.432+0000 7f51946db640 1 -- 172.21.15.100:0/555219232 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f518c06ee30 msgr2=0x7f518c1240a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:56:57.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.432+0000 7f51946db640 1 --2- 172.21.15.100:0/555219232 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f518c06ee30 0x7f518c1240a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5180000c00 tx=0x7f51800046c0 comp rx=0 tx=0).stop 2024-09-13T11:56:57.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.432+0000 7f51946db640 1 -- 172.21.15.100:0/555219232 shutdown_connections 2024-09-13T11:56:57.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.432+0000 7f51946db640 1 --2- 172.21.15.100:0/555219232 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f517803dfa0 0x7f5178040460 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:57.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.432+0000 7f51946db640 1 --2- 172.21.15.100:0/555219232 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f518c06ee30 0x7f518c1240a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:57.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.432+0000 7f51946db640 1 --2- 172.21.15.100:0/555219232 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f518c06f790 0x7f518c1245e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:57.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.432+0000 7f51946db640 1 --2- 172.21.15.100:0/555219232 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f518c06ea50 0x7f518c123b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:56:57.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.432+0000 7f51946db640 1 -- 172.21.15.100:0/555219232 >> 172.21.15.100:0/555219232 conn(0x7f518c0a68e0 msgr2=0x7f518c09fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:56:57.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.433+0000 7f51946db640 1 -- 172.21.15.100:0/555219232 shutdown_connections 2024-09-13T11:56:57.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:56:57.433+0000 7f51946db640 1 -- 172.21.15.100:0/555219232 wait complete. 2024-09-13T11:56:58.075 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:58.075 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37947]: starting. 2024-09-13T11:56:58.075 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37947]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-37911 2024-09-13T11:56:58.075 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[37947]: shutting down. 2024-09-13T11:56:58.076 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:58.077 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:56:58.077 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:56:58.080 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:57:00.165 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38015]: starting. 2024-09-13T11:57:00.165 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38015]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-38014 2024-09-13T11:57:00.166 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38015]: created object 0... 2024-09-13T11:57:00.166 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38015]: created object 25... 2024-09-13T11:57:00.166 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38015]: created object 49... 2024-09-13T11:57:00.166 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38015]: finishing. 2024-09-13T11:57:00.166 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38015]: shutting down. 2024-09-13T11:57:00.445 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:57:00.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.542+0000 7f192c879640 10 monclient: get_monmap_and_config 2024-09-13T11:57:00.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.542+0000 7f192c879640 10 monclient: build_initial_monmap 2024-09-13T11:57:00.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.542+0000 7f192c879640 10 monclient: monmap: 2024-09-13T11:57:00.543 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:00.543 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:00.544 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:57:00.542998+0000 2024-09-13T11:57:00.544 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:57:00.542998+0000 2024-09-13T11:57:00.544 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:00.544 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:00.544 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:57:00.544 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:57:00.544 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:57:00.544 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:57:00.544 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:57:00.544 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:00.552 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.542+0000 7f192c879640 1 Processor -- start 2024-09-13T11:57:00.552 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.542+0000 7f192c879640 1 -- start start 2024-09-13T11:57:00.552 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.542+0000 7f192c879640 10 monclient: init 2024-09-13T11:57:00.552 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f192c879640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:00.552 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f192c879640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-13T11:57:00.552 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f192c879640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1928137a60 addr v1:172.21.15.100:6789/0 2024-09-13T11:57:00.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f192c879640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19280a2980 0x7f192813c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:00.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f192c879640 10 monclient(hunting): picked mon.noname-b con 0x7f19280a2980 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:00.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f192c879640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f192813ce90 0x7f192813f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:00.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f192c879640 10 monclient(hunting): picked mon.noname-a con 0x7f192813ce90 addr v2:172.21.15.100:3300/0 2024-09-13T11:57:00.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f192c879640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1928138550 con 0x7f1928137a60 2024-09-13T11:57:00.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f192c879640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:00.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f192c879640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f19280a4410 con 0x7f192813ce90 2024-09-13T11:57:00.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f192c879640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:00.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f192c879640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f19280a5be0 con 0x7f19280a2980 2024-09-13T11:57:00.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f192c879640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:00.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f192c879640 10 monclient(hunting): authenticate will time out at 1001.136495s 2024-09-13T11:57:00.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1925d9b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19280a2980 0x7f192813c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:00.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f192659c640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f1928137a60 0x7f1928137e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:58924/0 (socket says 172.21.15.100:58924) 2024-09-13T11:57:00.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f192659c640 1 -- 172.21.15.100:0/3142755691 learned_addr learned my addr 172.21.15.100:0/3142755691 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:57:00.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1926d9d640 1 --2- 172.21.15.100:0/3142755691 >> v2:172.21.15.100:3300/0 conn(0x7f192813ce90 0x7f192813f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:00.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1925d9b640 10 monclient(hunting): get_auth_request con 0x7f19280a2980 auth_method 0 2024-09-13T11:57:00.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1925d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:00.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1925d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:00.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1925d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:00.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1926d9d640 10 monclient(hunting): get_auth_request con 0x7f192813ce90 auth_method 0 2024-09-13T11:57:00.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1926d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:00.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1926d9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:00.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1926d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:00.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1925d9b640 1 -- 172.21.15.100:0/3142755691 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19280a2980 msgr2=0x7f192813c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T11:57:00.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1925d9b640 1 -- 172.21.15.100:0/3142755691 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19280a2980 msgr2=0x7f192813c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:57:00.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1925d9b640 1 --2- 172.21.15.100:0/3142755691 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19280a2980 0x7f192813c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:57:00.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1926d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:00.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1926d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:00.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1926d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:00.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1925d9b640 1 --2- 172.21.15.100:0/3142755691 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19280a2980 0x7f192813c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:57:00.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1926d9d640 10 monclient(hunting): handle_auth_done global_id 14823 payload 274 2024-09-13T11:57:00.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1926d9d640 1 -- 172.21.15.100:0/3142755691 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19280a2980 msgr2=0x7f192813c950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:57:00.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1926d9d640 1 --2- 172.21.15.100:0/3142755691 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19280a2980 0x7f192813c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:00.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1926d9d640 1 -- 172.21.15.100:0/3142755691 >> v1:172.21.15.100:6789/0 conn(0x7f1928137a60 legacy=0x7f1928137e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:00.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1926d9d640 10 monclient: _finish_hunting 0 2024-09-13T11:57:00.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1926d9d640 1 monclient: found mon.noname-a 2024-09-13T11:57:00.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1926d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:00.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1926d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:57:00.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f192559a640 1 -- 172.21.15.100:0/3142755691 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2878298244 0 0) 0x7f1928138550 con 0x7f1928137a60 2024-09-13T11:57:00.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f1926d9d640 1 -- 172.21.15.100:0/3142755691 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19280a76b0 con 0x7f192813ce90 2024-09-13T11:57:00.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.543+0000 7f192559a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:57:00.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.544+0000 7f1926d9d640 1 --2- 172.21.15.100:0/3142755691 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f192813ce90 0x7f192813f280 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f19100097a0 tx=0x7f191002a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=612ed51f0e9d796 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:00.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.544+0000 7f192559a640 1 -- 172.21.15.100:0/3142755691 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1910037070 con 0x7f192813ce90 2024-09-13T11:57:00.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.544+0000 7f192559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:00.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.544+0000 7f192559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:57:00.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.544+0000 7f192559a640 10 monclient: dump: 2024-09-13T11:57:00.556 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:00.556 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:00.556 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:00.556 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:00.556 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:00.556 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:00.557 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:00.557 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:00.557 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:00.557 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:00.557 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.544+0000 7f192559a640 10 monclient: _finish_auth 0 2024-09-13T11:57:00.557 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.544+0000 7f192559a640 10 monclient: _check_auth_tickets 2024-09-13T11:57:00.557 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.544+0000 7f192559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:00.557 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.544+0000 7f192559a640 1 -- 172.21.15.100:0/3142755691 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f191002adb0 con 0x7f192813ce90 2024-09-13T11:57:00.557 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.544+0000 7f192c879640 5 monclient: authenticate success, global_id 14823 2024-09-13T11:57:00.557 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.544+0000 7f192c879640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:57:00.557 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.544+0000 7f192559a640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:00.557 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.544+0000 7f192c879640 10 monclient: get_monmap_and_config success 2024-09-13T11:57:00.557 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.544+0000 7f192559a640 1 -- 172.21.15.100:0/3142755691 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1910032660 con 0x7f192813ce90 2024-09-13T11:57:00.558 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.544+0000 7f192c879640 4 set_mon_vals no callback set 2024-09-13T11:57:00.558 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.544+0000 7f192559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:00.558 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.544+0000 7f192559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:00.558 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.544+0000 7f192559a640 10 monclient: dump: 2024-09-13T11:57:00.558 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:00.558 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:00.558 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:00.558 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:00.558 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:00.558 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:00.558 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:00.558 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:00.558 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:00.559 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:00.559 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.545+0000 7f192c879640 10 monclient: shutdown 2024-09-13T11:57:00.559 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.545+0000 7f192c879640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:00.559 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.545+0000 7f192c879640 1 -- 172.21.15.100:0/3142755691 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f192813ce90 msgr2=0x7f192813f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:00.559 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.545+0000 7f192c879640 1 --2- 172.21.15.100:0/3142755691 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f192813ce90 0x7f192813f280 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f19100097a0 tx=0x7f191002a060 comp rx=0 tx=0).stop 2024-09-13T11:57:00.559 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.545+0000 7f192c879640 1 -- 172.21.15.100:0/3142755691 shutdown_connections 2024-09-13T11:57:00.559 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.545+0000 7f192c879640 1 --2- 172.21.15.100:0/3142755691 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f192813ce90 0x7f192813f280 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:00.559 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.545+0000 7f192c879640 1 --2- 172.21.15.100:0/3142755691 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19280a2980 0x7f192813c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:00.559 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.545+0000 7f192c879640 1 -- 172.21.15.100:0/3142755691 >> 172.21.15.100:0/3142755691 conn(0x7f19280aa580 msgr2=0x7f19280a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:00.559 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.545+0000 7f192c879640 1 -- 172.21.15.100:0/3142755691 shutdown_connections 2024-09-13T11:57:00.559 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.545+0000 7f192c879640 1 -- 172.21.15.100:0/3142755691 wait complete. 2024-09-13T11:57:00.559 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.546+0000 7f192c879640 10 monclient: build_initial_monmap 2024-09-13T11:57:00.559 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.546+0000 7f192c879640 10 monclient: monmap: 2024-09-13T11:57:00.560 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:00.560 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:57:00.560 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:57:00.560 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:57:00.560 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:00.560 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:00.560 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:57:00.560 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:57:00.560 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:57:00.560 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:00.560 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.546+0000 7f192c879640 1 Processor -- start 2024-09-13T11:57:00.560 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.546+0000 7f192c879640 1 -- start start 2024-09-13T11:57:00.560 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.546+0000 7f192c879640 10 monclient: init 2024-09-13T11:57:00.561 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.546+0000 7f192c879640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:00.561 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.546+0000 7f192c879640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:57:00.561 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.546+0000 7f192c879640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19280a2980 0x7f1928136380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:00.561 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.546+0000 7f192c879640 10 monclient(hunting): picked mon.noname-a con 0x7f19280a2980 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:57:00.561 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.546+0000 7f192c879640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1928137a60 0x7f19281368c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:00.561 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.546+0000 7f192c879640 10 monclient(hunting): picked mon.noname-c con 0x7f1928137a60 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:00.561 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.546+0000 7f192c879640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f192813ce90 0x7f192809d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:00.561 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.546+0000 7f192c879640 10 monclient(hunting): picked mon.noname-b con 0x7f192813ce90 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:57:00.561 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.546+0000 7f192c879640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:00.561 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.546+0000 7f192c879640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f192806fd80 con 0x7f19280a2980 2024-09-13T11:57:00.561 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.546+0000 7f192c879640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:00.561 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.546+0000 7f192c879640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f19280a1700 con 0x7f192813ce90 2024-09-13T11:57:00.561 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.546+0000 7f192c879640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:00.561 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.546+0000 7f192c879640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f192806ff00 con 0x7f1928137a60 2024-09-13T11:57:00.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.546+0000 7f192c879640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:00.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.546+0000 7f192c879640 10 monclient(hunting): authenticate will time out at 1001.139910s 2024-09-13T11:57:00.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.546+0000 7f192659c640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19280a2980 0x7f1928136380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:00.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.546+0000 7f1925d9b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1928137a60 0x7f19281368c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:00.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192659c640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19280a2980 0x7f1928136380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47122/0 (socket says 172.21.15.100:47122) 2024-09-13T11:57:00.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f1925d9b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1928137a60 0x7f19281368c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:59064/0 (socket says 172.21.15.100:59064) 2024-09-13T11:57:00.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192659c640 1 -- 172.21.15.100:0/4128056429 learned_addr learned my addr 172.21.15.100:0/4128056429 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:00.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192659c640 10 monclient(hunting): get_auth_request con 0x7f19280a2980 auth_method 0 2024-09-13T11:57:00.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:00.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192659c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:00.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:00.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f1925d9b640 10 monclient(hunting): get_auth_request con 0x7f1928137a60 auth_method 0 2024-09-13T11:57:00.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f1925d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:00.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f1925d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:00.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f1925d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:00.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:00.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:00.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:00.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f1925d9b640 1 -- 172.21.15.100:0/4128056429 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1928137a60 msgr2=0x7f19281368c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-13T11:57:00.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f1925d9b640 1 -- 172.21.15.100:0/4128056429 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1928137a60 msgr2=0x7f19281368c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T11:57:00.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f1925d9b640 1 --2- 172.21.15.100:0/4128056429 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1928137a60 0x7f19281368c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:57:00.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f1925d9b640 1 --2- 172.21.15.100:0/4128056429 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1928137a60 0x7f19281368c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:57:00.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f1926d9d640 1 --2- 172.21.15.100:0/4128056429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f192813ce90 0x7f192809d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:00.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192659c640 10 monclient(hunting): handle_auth_done global_id 14829 payload 824 2024-09-13T11:57:00.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192659c640 1 -- 172.21.15.100:0/4128056429 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1928137a60 msgr2=0x7f19281368c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:57:00.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192659c640 1 --2- 172.21.15.100:0/4128056429 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1928137a60 0x7f19281368c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:00.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192659c640 1 -- 172.21.15.100:0/4128056429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f192813ce90 msgr2=0x7f192809d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:00.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192659c640 1 --2- 172.21.15.100:0/4128056429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f192813ce90 0x7f192809d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:00.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192659c640 10 monclient: _finish_hunting 0 2024-09-13T11:57:00.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192659c640 1 monclient: found mon.noname-a 2024-09-13T11:57:00.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:00.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:57:00.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192659c640 1 -- 172.21.15.100:0/4128056429 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1928120c00 con 0x7f19280a2980 2024-09-13T11:57:00.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f1926d9d640 10 monclient: get_auth_request con 0x7f192813ce90 auth_method 0 2024-09-13T11:57:00.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f1926d9d640 1 --2- 172.21.15.100:0/4128056429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f192813ce90 0x7f192809d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:57:00.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192659c640 1 --2- 172.21.15.100:0/4128056429 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19280a2980 0x7f1928136380 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f192000c490 tx=0x7f192000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:00.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f191effd640 1 -- 172.21.15.100:0/4128056429 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1920018070 con 0x7f19280a2980 2024-09-13T11:57:00.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f191effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:00.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f191effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:57:00.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f191effd640 10 monclient: dump: 2024-09-13T11:57:00.565 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:00.565 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:00.565 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:00.565 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:00.565 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:00.565 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:00.565 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:00.565 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:00.565 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:00.565 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:00.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f191effd640 10 monclient: _finish_auth 0 2024-09-13T11:57:00.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f191effd640 10 monclient: _check_auth_tickets 2024-09-13T11:57:00.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f191effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:00.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192c879640 5 monclient: authenticate success, global_id 14829 2024-09-13T11:57:00.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f191effd640 1 -- 172.21.15.100:0/4128056429 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1920007500 con 0x7f19280a2980 2024-09-13T11:57:00.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f191effd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:00.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f191effd640 1 -- 172.21.15.100:0/4128056429 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1920004a20 con 0x7f19280a2980 2024-09-13T11:57:00.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192c879640 10 monclient: _renew_subs 2024-09-13T11:57:00.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192c879640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:00.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192c879640 1 -- 172.21.15.100:0/4128056429 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f192809f5a0 con 0x7f19280a2980 2024-09-13T11:57:00.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f191effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:00.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f191effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:00.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f191effd640 10 monclient: dump: 2024-09-13T11:57:00.566 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:00.566 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:00.566 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:00.567 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:00.567 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:00.567 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:00.567 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:00.567 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:00.567 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:00.567 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:00.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f1924d99640 4 set_mon_vals no callback set 2024-09-13T11:57:00.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192c879640 10 monclient: _renew_subs 2024-09-13T11:57:00.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192c879640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:00.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.547+0000 7f192c879640 1 -- 172.21.15.100:0/4128056429 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f192809fc60 con 0x7f19280a2980 2024-09-13T11:57:00.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.548+0000 7f192c879640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:00.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.548+0000 7f192c879640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:00.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.548+0000 7f192c879640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:00.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.548+0000 7f191effd640 1 -- 172.21.15.100:0/4128056429 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f192001b070 con 0x7f19280a2980 2024-09-13T11:57:00.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.548+0000 7f192c879640 1 -- 172.21.15.100:0/4128056429 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1928137e40 con 0x7f19280a2980 2024-09-13T11:57:00.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.549+0000 7f191effd640 1 --2- 172.21.15.100:0/4128056429 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f190803dfa0 0x7f1908040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:00.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.549+0000 7f191effd640 1 -- 172.21.15.100:0/4128056429 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(219..219 src has 1..219) ==== 4738+0+0 (secure 0 0 0) 0x7f19200519d0 con 0x7f19280a2980 2024-09-13T11:57:00.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.549+0000 7f1925d9b640 1 --2- 172.21.15.100:0/4128056429 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f190803dfa0 0x7f1908040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:00.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.549+0000 7f1925d9b640 10 monclient: get_auth_request con 0x7f190803dfa0 auth_method 0 2024-09-13T11:57:00.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.549+0000 7f1925d9b640 1 --2- 172.21.15.100:0/4128056429 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f190803dfa0 0x7f1908040460 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1928137100 tx=0x7f191800b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:00.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.553+0000 7f191effd640 1 -- 172.21.15.100:0/4128056429 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f1920004da0 con 0x7f19280a2980 2024-09-13T11:57:00.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.553+0000 7f191effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:00.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.553+0000 7f191effd640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:57:00.798 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.797+0000 7f192c879640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:57:00.798 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.797+0000 7f192c879640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:57:00.798 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.797+0000 7f192c879640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:00.799 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.797+0000 7f192c879640 1 -- 172.21.15.100:0/4128056429 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f19280a00a0 con 0x7f19280a2980 2024-09-13T11:57:00.799 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.797+0000 7f191effd640 1 -- 172.21.15.100:0/4128056429 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f192001bd70 con 0x7f19280a2980 2024-09-13T11:57:00.799 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.797+0000 7f191effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:57:00.799 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.797+0000 7f191effd640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:57:00.799 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:57:00.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.799+0000 7f192c879640 1 -- 172.21.15.100:0/4128056429 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f190803dfa0 msgr2=0x7f1908040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:00.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.799+0000 7f192c879640 1 --2- 172.21.15.100:0/4128056429 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f190803dfa0 0x7f1908040460 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1928137100 tx=0x7f191800b040 comp rx=0 tx=0).stop 2024-09-13T11:57:00.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.799+0000 7f192c879640 10 monclient: shutdown 2024-09-13T11:57:00.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.799+0000 7f192c879640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:00.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.799+0000 7f192c879640 1 -- 172.21.15.100:0/4128056429 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19280a2980 msgr2=0x7f1928136380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:00.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.799+0000 7f192c879640 1 --2- 172.21.15.100:0/4128056429 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19280a2980 0x7f1928136380 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f192000c490 tx=0x7f192000c950 comp rx=0 tx=0).stop 2024-09-13T11:57:00.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.799+0000 7f192c879640 1 -- 172.21.15.100:0/4128056429 shutdown_connections 2024-09-13T11:57:00.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.799+0000 7f192c879640 1 --2- 172.21.15.100:0/4128056429 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f190803dfa0 0x7f1908040460 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:00.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.799+0000 7f192c879640 1 --2- 172.21.15.100:0/4128056429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f192813ce90 0x7f192809d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:00.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.799+0000 7f192c879640 1 --2- 172.21.15.100:0/4128056429 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1928137a60 0x7f19281368c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:00.802 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.799+0000 7f192c879640 1 --2- 172.21.15.100:0/4128056429 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19280a2980 0x7f1928136380 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:00.802 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.799+0000 7f192c879640 1 -- 172.21.15.100:0/4128056429 >> 172.21.15.100:0/4128056429 conn(0x7f19280aa580 msgr2=0x7f192813d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:00.802 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.799+0000 7f192c879640 1 -- 172.21.15.100:0/4128056429 shutdown_connections 2024-09-13T11:57:00.802 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.799+0000 7f192c879640 1 -- 172.21.15.100:0/4128056429 wait complete. 2024-09-13T11:57:00.811 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-13T11:57:00.811 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-13T11:57:00.811 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-13T11:57:00.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.907+0000 7fb44f850640 10 monclient: get_monmap_and_config 2024-09-13T11:57:00.909 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.907+0000 7fb44f850640 10 monclient: build_initial_monmap 2024-09-13T11:57:00.909 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.907+0000 7fb44f850640 10 monclient: monmap: 2024-09-13T11:57:00.909 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:00.909 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:00.909 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:57:00.908503+0000 2024-09-13T11:57:00.909 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:57:00.908503+0000 2024-09-13T11:57:00.909 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:00.909 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:00.909 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:57:00.909 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:57:00.909 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:57:00.909 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:57:00.909 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:57:00.910 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:00.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.908+0000 7fb44f850640 1 Processor -- start 2024-09-13T11:57:00.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.908+0000 7fb44f850640 1 -- start start 2024-09-13T11:57:00.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.908+0000 7fb44f850640 10 monclient: init 2024-09-13T11:57:00.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.908+0000 7fb44f850640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:00.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.908+0000 7fb44f850640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-13T11:57:00.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.908+0000 7fb44f850640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb4480a1d80 addr v1:172.21.15.122:6789/0 2024-09-13T11:57:00.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.908+0000 7fb44f850640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb4480a2160 0x7fb4480a2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:00.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.908+0000 7fb44f850640 10 monclient(hunting): picked mon.noname-a con 0x7fb4480a2160 addr v2:172.21.15.100:3300/0 2024-09-13T11:57:00.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.908+0000 7fb44f850640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb44813c9b0 0x7fb44813eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:00.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.908+0000 7fb44f850640 10 monclient(hunting): picked mon.noname-c con 0x7fb44813c9b0 addr v2:172.21.15.122:3300/0 2024-09-13T11:57:00.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.908+0000 7fb44f850640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb44806d8c0 con 0x7fb4480a1d80 2024-09-13T11:57:00.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.908+0000 7fb44f850640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:00.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.908+0000 7fb44f850640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7fb44809a6b0 con 0x7fb4480a2160 2024-09-13T11:57:00.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.908+0000 7fb44f850640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:00.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.908+0000 7fb44f850640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7fb44809ac70 con 0x7fb44813c9b0 2024-09-13T11:57:00.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.908+0000 7fb44f850640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:00.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.908+0000 7fb44f850640 10 monclient(hunting): authenticate will time out at 1001.501912s 2024-09-13T11:57:00.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44cdeb640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb4480a2160 0x7fb4480a2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:00.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44cdeb640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb4480a2160 0x7fb4480a2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47128/0 (socket says 172.21.15.100:47128) 2024-09-13T11:57:00.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44cdeb640 1 -- 172.21.15.100:0/2353824582 learned_addr learned my addr 172.21.15.100:0/2353824582 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:00.920 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44cdeb640 10 monclient(hunting): get_auth_request con 0x7fb4480a2160 auth_method 0 2024-09-13T11:57:00.920 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44cdeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:00.920 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44cdeb640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:00.920 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44cdeb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:00.920 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44cdeb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:00.920 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44cdeb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:00.920 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44dded640 1 --2- 172.21.15.100:0/2353824582 >> v2:172.21.15.122:3300/0 conn(0x7fb44813c9b0 0x7fb44813eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:00.920 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44cdeb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:00.920 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44dded640 10 monclient(hunting): get_auth_request con 0x7fb44813c9b0 auth_method 0 2024-09-13T11:57:00.920 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44dded640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:00.920 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44dded640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:00.920 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44dded640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:00.920 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44dded640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:00.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44dded640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:00.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44dded640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:00.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44cdeb640 10 monclient(hunting): handle_auth_done global_id 14832 payload 274 2024-09-13T11:57:00.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44cdeb640 1 -- 172.21.15.100:0/2353824582 >> v2:172.21.15.122:3300/0 conn(0x7fb44813c9b0 msgr2=0x7fb44813eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:57:00.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44cdeb640 1 --2- 172.21.15.100:0/2353824582 >> v2:172.21.15.122:3300/0 conn(0x7fb44813c9b0 0x7fb44813eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:00.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44cdeb640 1 -- 172.21.15.100:0/2353824582 >> v1:172.21.15.122:6789/0 conn(0x7fb4480a1d80 legacy=0x7fb44809b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:00.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44cdeb640 10 monclient: _finish_hunting 0 2024-09-13T11:57:00.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44cdeb640 1 monclient: found mon.noname-a 2024-09-13T11:57:00.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44cdeb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:00.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44cdeb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:57:00.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.909+0000 7fb44cdeb640 1 -- 172.21.15.100:0/2353824582 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4480a2ab0 con 0x7fb4480a2160 2024-09-13T11:57:00.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb44cdeb640 1 --2- 172.21.15.100:0/2353824582 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4480a2160 0x7fb4480a2570 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb4480a3c90 tx=0x7fb43802a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec9e648b734037ac server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:00.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb43ffff640 1 -- 172.21.15.100:0/2353824582 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb438037070 con 0x7fb4480a2160 2024-09-13T11:57:00.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb43ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:00.922 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb43ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:57:00.922 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb43ffff640 10 monclient: dump: 2024-09-13T11:57:00.922 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:00.922 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:00.922 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:00.922 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:00.922 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:00.922 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:00.922 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:00.922 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:00.922 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:00.922 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:00.922 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb43ffff640 10 monclient: _finish_auth 0 2024-09-13T11:57:00.922 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb43ffff640 10 monclient: _check_auth_tickets 2024-09-13T11:57:00.923 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb43ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:00.923 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb43ffff640 1 -- 172.21.15.100:0/2353824582 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb438032440 con 0x7fb4480a2160 2024-09-13T11:57:00.923 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb44f850640 5 monclient: authenticate success, global_id 14832 2024-09-13T11:57:00.923 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb44f850640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:57:00.923 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb43ffff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:00.923 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb44f850640 10 monclient: get_monmap_and_config success 2024-09-13T11:57:00.923 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb43ffff640 1 -- 172.21.15.100:0/2353824582 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb438032810 con 0x7fb4480a2160 2024-09-13T11:57:00.923 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb44f850640 4 set_mon_vals no callback set 2024-09-13T11:57:00.923 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb43ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:00.923 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb43ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:00.923 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb43ffff640 10 monclient: dump: 2024-09-13T11:57:00.923 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:00.923 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:00.923 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:00.924 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:00.924 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:00.924 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:00.924 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:00.924 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:00.924 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:00.924 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:00.924 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb44f850640 10 monclient: shutdown 2024-09-13T11:57:00.924 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb44f850640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:00.924 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb44f850640 1 -- 172.21.15.100:0/2353824582 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4480a2160 msgr2=0x7fb4480a2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:00.924 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb44f850640 1 --2- 172.21.15.100:0/2353824582 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4480a2160 0x7fb4480a2570 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb4480a3c90 tx=0x7fb43802a150 comp rx=0 tx=0).stop 2024-09-13T11:57:00.924 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb44f850640 1 -- 172.21.15.100:0/2353824582 shutdown_connections 2024-09-13T11:57:00.924 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb44f850640 1 --2- 172.21.15.100:0/2353824582 >> v2:172.21.15.122:3300/0 conn(0x7fb44813c9b0 0x7fb44813eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:00.925 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb44f850640 1 --2- 172.21.15.100:0/2353824582 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4480a2160 0x7fb4480a2570 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:00.925 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb44f850640 1 -- 172.21.15.100:0/2353824582 >> 172.21.15.100:0/2353824582 conn(0x7fb4480a2e70 msgr2=0x7fb4480a3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:00.925 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb44f850640 1 -- 172.21.15.100:0/2353824582 shutdown_connections 2024-09-13T11:57:00.925 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.910+0000 7fb44f850640 1 -- 172.21.15.100:0/2353824582 wait complete. 2024-09-13T11:57:00.925 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.911+0000 7fb44f850640 10 monclient: build_initial_monmap 2024-09-13T11:57:00.925 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.911+0000 7fb44f850640 10 monclient: monmap: 2024-09-13T11:57:00.925 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:00.925 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:57:00.925 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:57:00.925 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:57:00.925 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:00.925 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:00.925 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:57:00.925 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:57:00.926 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:57:00.926 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:00.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.911+0000 7fb44f850640 1 Processor -- start 2024-09-13T11:57:00.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.911+0000 7fb44f850640 1 -- start start 2024-09-13T11:57:00.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.911+0000 7fb44f850640 10 monclient: init 2024-09-13T11:57:00.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44f850640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:00.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44f850640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:57:00.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44f850640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4480a1d80 0x7fb4481cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:00.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44f850640 10 monclient(hunting): picked mon.noname-b con 0x7fb4480a1d80 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:57:00.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44f850640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4480a2160 0x7fb4481cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:00.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44f850640 10 monclient(hunting): picked mon.noname-c con 0x7fb4480a2160 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:00.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44f850640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb44813c9b0 0x7fb4481cd360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:00.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44f850640 10 monclient(hunting): picked mon.noname-a con 0x7fb44813c9b0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:57:00.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44f850640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:00.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44f850640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fb44813f460 con 0x7fb44813c9b0 2024-09-13T11:57:00.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44f850640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:00.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44f850640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb44813f2e0 con 0x7fb4480a1d80 2024-09-13T11:57:00.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44f850640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:00.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44f850640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fb44813f5e0 con 0x7fb4480a2160 2024-09-13T11:57:00.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44f850640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:00.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44f850640 10 monclient(hunting): authenticate will time out at 1001.505285s 2024-09-13T11:57:00.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44cdeb640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4480a2160 0x7fb4481cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:00.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44dded640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb44813c9b0 0x7fb4481cd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:00.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44cdeb640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4480a2160 0x7fb4481cce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:59066/0 (socket says 172.21.15.100:59066) 2024-09-13T11:57:00.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44cdeb640 1 -- 172.21.15.100:0/2922813010 learned_addr learned my addr 172.21.15.100:0/2922813010 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:00.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44cdeb640 10 monclient(hunting): get_auth_request con 0x7fb4480a2160 auth_method 0 2024-09-13T11:57:00.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44cdeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:00.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44cdeb640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:00.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44cdeb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:00.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44dded640 10 monclient(hunting): get_auth_request con 0x7fb44813c9b0 auth_method 0 2024-09-13T11:57:00.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44dded640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:00.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44dded640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:00.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44dded640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:00.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44cdeb640 1 -- 172.21.15.100:0/2922813010 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4480a2160 msgr2=0x7fb4481cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-13T11:57:00.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44dded640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:00.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44dded640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:00.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44cdeb640 1 -- 172.21.15.100:0/2922813010 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4480a2160 msgr2=0x7fb4481cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T11:57:00.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44cdeb640 1 --2- 172.21.15.100:0/2922813010 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4480a2160 0x7fb4481cce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:57:00.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44dded640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:00.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44cdeb640 1 --2- 172.21.15.100:0/2922813010 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4480a2160 0x7fb4481cce20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:57:00.929 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44d5ec640 1 --2- 172.21.15.100:0/2922813010 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4480a1d80 0x7fb4481cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:00.929 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44d5ec640 10 monclient(hunting): get_auth_request con 0x7fb4480a1d80 auth_method 0 2024-09-13T11:57:00.929 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44d5ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:00.929 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44d5ec640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:00.929 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44d5ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:00.929 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44dded640 10 monclient(hunting): handle_auth_done global_id 14835 payload 824 2024-09-13T11:57:00.929 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44dded640 1 -- 172.21.15.100:0/2922813010 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4480a2160 msgr2=0x7fb4481cce20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:57:00.929 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44dded640 1 --2- 172.21.15.100:0/2922813010 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4480a2160 0x7fb4481cce20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:00.929 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44dded640 1 -- 172.21.15.100:0/2922813010 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4480a1d80 msgr2=0x7fb4481cc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:00.929 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44dded640 1 --2- 172.21.15.100:0/2922813010 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4480a1d80 0x7fb4481cc8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:00.929 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44dded640 10 monclient: _finish_hunting 0 2024-09-13T11:57:00.929 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44dded640 1 monclient: found mon.noname-a 2024-09-13T11:57:00.929 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44dded640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:00.929 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44dded640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:57:00.930 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44dded640 1 -- 172.21.15.100:0/2922813010 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb44808ffc0 con 0x7fb44813c9b0 2024-09-13T11:57:00.930 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb44dded640 1 --2- 172.21.15.100:0/2922813010 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb44813c9b0 0x7fb4481cd360 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb44000ef90 tx=0x7fb44000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:00.930 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.912+0000 7fb43dffb640 1 -- 172.21.15.100:0/2922813010 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb440018020 con 0x7fb44813c9b0 2024-09-13T11:57:00.930 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.913+0000 7fb43dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:00.930 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.913+0000 7fb43dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:57:00.930 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.913+0000 7fb43dffb640 10 monclient: dump: 2024-09-13T11:57:00.930 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:00.930 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:00.930 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:00.930 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:00.930 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:00.930 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:00.930 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:00.930 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:00.931 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:00.931 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:00.931 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.913+0000 7fb43dffb640 10 monclient: _finish_auth 0 2024-09-13T11:57:00.931 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.913+0000 7fb43dffb640 10 monclient: _check_auth_tickets 2024-09-13T11:57:00.931 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.913+0000 7fb43dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:00.931 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.913+0000 7fb44f850640 5 monclient: authenticate success, global_id 14835 2024-09-13T11:57:00.931 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.913+0000 7fb43dffb640 1 -- 172.21.15.100:0/2922813010 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb440014480 con 0x7fb44813c9b0 2024-09-13T11:57:00.931 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.913+0000 7fb44f850640 10 monclient: _renew_subs 2024-09-13T11:57:00.931 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.913+0000 7fb44f850640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:00.931 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.913+0000 7fb44f850640 1 -- 172.21.15.100:0/2922813010 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4480902a0 con 0x7fb44813c9b0 2024-09-13T11:57:00.931 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.913+0000 7fb43dffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:00.931 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.913+0000 7fb43dffb640 1 -- 172.21.15.100:0/2922813010 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb440014870 con 0x7fb44813c9b0 2024-09-13T11:57:00.931 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.913+0000 7fb43dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:00.931 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.913+0000 7fb43dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:00.932 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.913+0000 7fb43dffb640 10 monclient: dump: 2024-09-13T11:57:00.932 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:00.932 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:00.932 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:00.932 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:00.932 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:00.932 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:00.932 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:00.932 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:00.932 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:00.932 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:00.932 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.913+0000 7fb43f7fe640 4 set_mon_vals no callback set 2024-09-13T11:57:00.932 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.913+0000 7fb44f850640 10 monclient: _renew_subs 2024-09-13T11:57:00.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.913+0000 7fb44f850640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:00.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.913+0000 7fb44f850640 1 -- 172.21.15.100:0/2922813010 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb44813ef90 con 0x7fb44813c9b0 2024-09-13T11:57:00.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.913+0000 7fb44f850640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:00.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.913+0000 7fb44f850640 1 --2- 172.21.15.100:0/2922813010 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4480a0250 0x7fb448137cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:00.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.913+0000 7fb44f850640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:00.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.913+0000 7fb44f850640 1 -- 172.21.15.100:0/2922813010 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fb44808eb00 con 0x7fb4480a0250 2024-09-13T11:57:00.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb43dffb640 1 -- 172.21.15.100:0/2922813010 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fb440005d10 con 0x7fb44813c9b0 2024-09-13T11:57:00.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb44d5ec640 1 --2- 172.21.15.100:0/2922813010 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4480a0250 0x7fb448137cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:00.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb44d5ec640 10 monclient: get_auth_request con 0x7fb4480a0250 auth_method 0 2024-09-13T11:57:00.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb44d5ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:00.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb44d5ec640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:00.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb44d5ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:00.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb44d5ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:00.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb44d5ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:00.934 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb44d5ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:00.934 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb43dffb640 1 --2- 172.21.15.100:0/2922813010 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fb42c03dfd0 0x7fb42c040490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:00.934 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb43dffb640 1 -- 172.21.15.100:0/2922813010 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(219..219 src has 1..219) ==== 4738+0+0 (secure 0 0 0) 0x7fb440053640 con 0x7fb44813c9b0 2024-09-13T11:57:00.934 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb44d5ec640 10 monclient(hunting): handle_auth_done global_id 14838 payload 824 2024-09-13T11:57:00.934 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb44d5ec640 1 -- 172.21.15.100:0/2922813010 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb4480a2570 con 0x7fb4480a0250 2024-09-13T11:57:00.934 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb44d5ec640 1 --2- 172.21.15.100:0/2922813010 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4480a0250 0x7fb448137cb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb44400e500 tx=0x7fb44400e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:00.934 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb43dffb640 1 -- 172.21.15.100:0/2922813010 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4440052e0 con 0x7fb4480a0250 2024-09-13T11:57:00.934 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb43dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:00.934 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb43dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:00.934 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb43dffb640 10 monclient: dump: 2024-09-13T11:57:00.934 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:00.934 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:00.934 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:00.934 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:00.935 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:00.935 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:00.935 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:00.935 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:00.935 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:00.935 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:00.935 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb44cdeb640 1 --2- 172.21.15.100:0/2922813010 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fb42c03dfd0 0x7fb42c040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:00.935 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb44cdeb640 10 monclient: get_auth_request con 0x7fb42c03dfd0 auth_method 0 2024-09-13T11:57:00.935 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb43dffb640 1 -- 172.21.15.100:0/2922813010 <== mon.0 v2:172.21.15.100:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb444018070 con 0x7fb4480a0250 2024-09-13T11:57:00.935 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb43dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:00.935 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb43dffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:57:00.935 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb43dffb640 1 -- 172.21.15.100:0/2922813010 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4480a0250 msgr2=0x7fb448137cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:00.935 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb43dffb640 1 --2- 172.21.15.100:0/2922813010 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4480a0250 0x7fb448137cb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb44400e500 tx=0x7fb44400e9c0 comp rx=0 tx=0).stop 2024-09-13T11:57:00.935 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb43dffb640 1 -- 172.21.15.100:0/2922813010 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4480a0250 msgr2=0x7fb448137cb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:57:00.936 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.914+0000 7fb43dffb640 1 --2- 172.21.15.100:0/2922813010 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4480a0250 0x7fb448137cb0 secure :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb44400e500 tx=0x7fb44400e9c0 comp rx=0 tx=0).stop 2024-09-13T11:57:00.936 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.915+0000 7fb44cdeb640 1 --2- 172.21.15.100:0/2922813010 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fb42c03dfd0 0x7fb42c040490 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb44808e580 tx=0x7fb43802e000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:00.937 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.920+0000 7fb44f850640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:57:00.937 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.920+0000 7fb44f850640 1 --2- 172.21.15.100:0/2922813010 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb44802d1f0 0x7fb44802d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:00.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.920+0000 7fb44f850640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:00.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.920+0000 7fb44f850640 1 -- 172.21.15.100:0/2922813010 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fb44813b0b0 con 0x7fb44802d1f0 2024-09-13T11:57:00.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.921+0000 7fb44d5ec640 1 --2- 172.21.15.100:0/2922813010 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb44802d1f0 0x7fb44802d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:00.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.921+0000 7fb44d5ec640 10 monclient: get_auth_request con 0x7fb44802d1f0 auth_method 0 2024-09-13T11:57:00.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.921+0000 7fb44d5ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:00.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.921+0000 7fb44d5ec640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:00.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.921+0000 7fb44d5ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:00.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.921+0000 7fb44d5ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:00.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.921+0000 7fb44d5ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:00.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.921+0000 7fb44d5ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:00.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.921+0000 7fb44d5ec640 10 monclient(hunting): handle_auth_done global_id 14841 payload 824 2024-09-13T11:57:00.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.921+0000 7fb44d5ec640 1 -- 172.21.15.100:0/2922813010 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb448069fa0 con 0x7fb44802d1f0 2024-09-13T11:57:00.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.921+0000 7fb44d5ec640 1 --2- 172.21.15.100:0/2922813010 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb44802d1f0 0x7fb44802d600 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb44400e500 tx=0x7fb444000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:00.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.921+0000 7fb43dffb640 1 -- 172.21.15.100:0/2922813010 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb444006be0 con 0x7fb44802d1f0 2024-09-13T11:57:00.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.921+0000 7fb43dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:00.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.921+0000 7fb43dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:00.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.921+0000 7fb43dffb640 10 monclient: dump: 2024-09-13T11:57:00.939 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:00.939 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:00.939 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:00.939 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:00.939 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:00.939 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:00.939 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:00.939 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:00.939 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:00.939 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:00.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.921+0000 7fb43dffb640 1 -- 172.21.15.100:0/2922813010 <== mon.0 v2:172.21.15.100:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7fb444006be0 con 0x7fb44802d1f0 2024-09-13T11:57:00.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.921+0000 7fb43dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:57:00.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.921+0000 7fb43dffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:57:00.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.921+0000 7fb43dffb640 1 -- 172.21.15.100:0/2922813010 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb44802d1f0 msgr2=0x7fb44802d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:00.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.921+0000 7fb43dffb640 1 --2- 172.21.15.100:0/2922813010 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb44802d1f0 0x7fb44802d600 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb44400e500 tx=0x7fb444000ed0 comp rx=0 tx=0).stop 2024-09-13T11:57:00.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.921+0000 7fb43dffb640 1 -- 172.21.15.100:0/2922813010 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb44802d1f0 msgr2=0x7fb44802d600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:57:00.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.921+0000 7fb43dffb640 1 --2- 172.21.15.100:0/2922813010 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb44802d1f0 0x7fb44802d600 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:00.940 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:57:00.940 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:57:00.940 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:57:00.941 INFO:teuthology.orchestra.run.smithi100.stdout: "state": "leader", 2024-09-13T11:57:00.941 INFO:teuthology.orchestra.run.smithi100.stdout: "election_epoch": 22, 2024-09-13T11:57:00.941 INFO:teuthology.orchestra.run.smithi100.stdout: "uptime": 121920, 2024-09-13T11:57:00.941 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum": [ 2024-09-13T11:57:00.941 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-13T11:57:00.941 INFO:teuthology.orchestra.run.smithi100.stdout: 1, 2024-09-13T11:57:00.941 INFO:teuthology.orchestra.run.smithi100.stdout: 2 2024-09-13T11:57:00.941 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:00.941 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_age": 3, 2024-09-13T11:57:00.941 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:57:00.941 INFO:teuthology.orchestra.run.smithi100.stdout: "required_con": "2450521708007866372", 2024-09-13T11:57:00.941 INFO:teuthology.orchestra.run.smithi100.stdout: "required_mon": [ 2024-09-13T11:57:00.941 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:57:00.941 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:57:00.942 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:57:00.942 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:57:00.942 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:57:00.942 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:57:00.942 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:57:00.942 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:57:00.942 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:57:00.942 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:57:00.942 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:57:00.942 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:00.942 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:57:00.942 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_mon": [ 2024-09-13T11:57:00.942 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:57:00.942 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:57:00.943 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:57:00.943 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:57:00.943 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:57:00.943 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:57:00.943 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:57:00.943 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:57:00.943 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:57:00.943 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:57:00.943 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:57:00.944 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:00.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:00.944 INFO:teuthology.orchestra.run.smithi100.stdout: "outside_quorum": [], 2024-09-13T11:57:00.944 INFO:teuthology.orchestra.run.smithi100.stdout: "extra_probe_peers": [], 2024-09-13T11:57:00.944 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_provider": [], 2024-09-13T11:57:00.944 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": { 2024-09-13T11:57:00.944 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-09-13T11:57:00.944 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "0b1f4b4d-fad4-4dd3-86ac-a286d80846c0", 2024-09-13T11:57:00.944 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:57:00.944 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:57:00.944 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release": 19, 2024-09-13T11:57:00.945 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release_name": "squid", 2024-09-13T11:57:00.945 INFO:teuthology.orchestra.run.smithi100.stdout: "election_strategy": 1, 2024-09-13T11:57:00.945 INFO:teuthology.orchestra.run.smithi100.stdout: "disallowed_leaders: ": "", 2024-09-13T11:57:00.945 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false, 2024-09-13T11:57:00.945 INFO:teuthology.orchestra.run.smithi100.stdout: "tiebreaker_mon": "", 2024-09-13T11:57:00.945 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_ranks: ": "", 2024-09-13T11:57:00.945 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:57:00.945 INFO:teuthology.orchestra.run.smithi100.stdout: "persistent": [ 2024-09-13T11:57:00.945 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:57:00.945 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:57:00.945 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:57:00.945 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:57:00.945 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:57:00.945 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:57:00.946 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:57:00.946 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:57:00.946 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:57:00.946 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:57:00.946 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:57:00.946 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:00.946 INFO:teuthology.orchestra.run.smithi100.stdout: "optional": [] 2024-09-13T11:57:00.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:00.946 INFO:teuthology.orchestra.run.smithi100.stdout: "mons": [ 2024-09-13T11:57:00.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:00.946 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:57:00.946 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:57:00.946 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:57:00.946 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:57:00.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:00.947 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:57:00.947 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3300", 2024-09-13T11:57:00.947 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:00.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:00.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:00.947 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:57:00.947 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789", 2024-09-13T11:57:00.947 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:00.947 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:00.948 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:00.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:00.948 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789/0", 2024-09-13T11:57:00.948 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-13T11:57:00.948 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:57:00.948 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:57:00.948 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:57:00.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:00.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:00.948 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:57:00.948 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:57:00.948 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:57:00.948 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:57:00.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:00.949 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:57:00.949 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:3300", 2024-09-13T11:57:00.949 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:00.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:00.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:00.949 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:57:00.949 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789", 2024-09-13T11:57:00.949 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:00.949 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:00.949 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:00.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:00.949 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789/0", 2024-09-13T11:57:00.950 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.122:6789/0", 2024-09-13T11:57:00.950 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:57:00.950 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:57:00.950 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:57:00.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:00.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:00.950 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:57:00.950 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:57:00.950 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:57:00.950 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:57:00.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:00.951 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:57:00.951 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3301", 2024-09-13T11:57:00.951 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:00.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:00.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:00.951 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:57:00.951 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790", 2024-09-13T11:57:00.951 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:00.951 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:00.951 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:00.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:00.951 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790/0", 2024-09-13T11:57:00.951 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-13T11:57:00.952 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:57:00.952 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:57:00.952 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:57:00.952 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:00.952 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:00.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:00.952 INFO:teuthology.orchestra.run.smithi100.stdout: "feature_map": { 2024-09-13T11:57:00.952 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": [ 2024-09-13T11:57:00.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:00.952 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:00.952 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:00.952 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:57:00.952 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:00.952 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:00.953 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": [ 2024-09-13T11:57:00.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:00.953 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:00.953 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:00.953 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 2 2024-09-13T11:57:00.953 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:00.953 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:00.953 INFO:teuthology.orchestra.run.smithi100.stdout: "client": [ 2024-09-13T11:57:00.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:00.953 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:00.953 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:00.953 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 2 2024-09-13T11:57:00.953 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:00.953 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:00.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:00.954 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false 2024-09-13T11:57:00.954 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:57:00.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.922+0000 7fb44f850640 1 -- 172.21.15.100:0/2922813010 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fb42c03dfd0 msgr2=0x7fb42c040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:00.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.922+0000 7fb44f850640 1 --2- 172.21.15.100:0/2922813010 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fb42c03dfd0 0x7fb42c040490 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb44808e580 tx=0x7fb43802e000 comp rx=0 tx=0).stop 2024-09-13T11:57:00.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.922+0000 7fb44f850640 10 monclient: shutdown 2024-09-13T11:57:00.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.922+0000 7fb44f850640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:00.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.922+0000 7fb44f850640 1 -- 172.21.15.100:0/2922813010 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb44813c9b0 msgr2=0x7fb4481cd360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:00.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.922+0000 7fb44f850640 1 --2- 172.21.15.100:0/2922813010 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb44813c9b0 0x7fb4481cd360 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb44000ef90 tx=0x7fb44000c490 comp rx=0 tx=0).stop 2024-09-13T11:57:00.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.922+0000 7fb44dded640 1 -- 172.21.15.100:0/2922813010 reap_dead start 2024-09-13T11:57:00.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.922+0000 7fb44f850640 1 -- 172.21.15.100:0/2922813010 shutdown_connections 2024-09-13T11:57:00.956 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.922+0000 7fb44f850640 1 --2- 172.21.15.100:0/2922813010 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb44813c9b0 0x7fb4481cd360 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:00.956 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.922+0000 7fb44f850640 1 -- 172.21.15.100:0/2922813010 >> 172.21.15.100:0/2922813010 conn(0x7fb4480a2e70 msgr2=0x7fb44813bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:00.956 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.922+0000 7fb44f850640 1 -- 172.21.15.100:0/2922813010 shutdown_connections 2024-09-13T11:57:00.956 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:00.922+0000 7fb44f850640 1 -- 172.21.15.100:0/2922813010 wait complete. 2024-09-13T11:57:00.956 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T11:57:00.956 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-13T11:57:00.956 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T11:57:01.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.053+0000 7f8d7b016640 10 monclient: get_monmap_and_config 2024-09-13T11:57:01.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.053+0000 7f8d7b016640 10 monclient: build_initial_monmap 2024-09-13T11:57:01.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.053+0000 7f8d7b016640 10 monclient: monmap: 2024-09-13T11:57:01.054 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:01.054 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:01.055 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:57:01.053964+0000 2024-09-13T11:57:01.055 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:57:01.053964+0000 2024-09-13T11:57:01.055 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:01.055 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:01.055 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:57:01.055 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:57:01.055 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:57:01.055 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:57:01.055 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:57:01.055 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:01.064 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.053+0000 7f8d7b016640 1 Processor -- start 2024-09-13T11:57:01.064 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.053+0000 7f8d7b016640 1 -- start start 2024-09-13T11:57:01.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.053+0000 7f8d7b016640 10 monclient: init 2024-09-13T11:57:01.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d7b016640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:01.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d7b016640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-13T11:57:01.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d7b016640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8d7406ea50 0x7f8d740a2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:01.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d7b016640 10 monclient(hunting): picked mon.noname-c con 0x7f8d7406ea50 addr v2:172.21.15.122:3300/0 2024-09-13T11:57:01.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d7b016640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7406ee30 0x7f8d7406f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:01.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d7b016640 10 monclient(hunting): picked mon.noname-b con 0x7f8d7406ee30 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:01.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d7b016640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8d7406f790 addr v1:172.21.15.122:6789/0 2024-09-13T11:57:01.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d7b016640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d740a1590 con 0x7f8d7406f790 2024-09-13T11:57:01.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d7b016640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:01.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d7b016640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f8d740a1870 con 0x7f8d7406ea50 2024-09-13T11:57:01.065 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d7b016640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:01.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d7b016640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f8d740a1b50 con 0x7f8d7406ee30 2024-09-13T11:57:01.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d7b016640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:01.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d7b016640 10 monclient(hunting): authenticate will time out at 1001.647358s 2024-09-13T11:57:01.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d73fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7406ee30 0x7f8d7406f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:01.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d73fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7406ee30 0x7f8d7406f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:59076/0 (socket says 172.21.15.100:59076) 2024-09-13T11:57:01.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d73fff640 1 -- 172.21.15.100:0/60911677 learned_addr learned my addr 172.21.15.100:0/60911677 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:01.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d73fff640 10 monclient(hunting): get_auth_request con 0x7f8d7406ee30 auth_method 0 2024-09-13T11:57:01.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:01.066 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d73fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:01.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d73fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:01.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d73fff640 1 -- 172.21.15.100:0/60911677 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7406ee30 msgr2=0x7f8d7406f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T11:57:01.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d73fff640 1 -- 172.21.15.100:0/60911677 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7406ee30 msgr2=0x7f8d7406f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:57:01.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d73fff640 1 --2- 172.21.15.100:0/60911677 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7406ee30 0x7f8d7406f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:57:01.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d73fff640 1 --2- 172.21.15.100:0/60911677 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7406ee30 0x7f8d7406f250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:57:01.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d78db2640 1 --2- 172.21.15.100:0/60911677 >> v2:172.21.15.122:3300/0 conn(0x7f8d7406ea50 0x7f8d740a2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:01.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d78db2640 10 monclient(hunting): get_auth_request con 0x7f8d7406ea50 auth_method 0 2024-09-13T11:57:01.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d78db2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:01.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d78db2640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:01.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d78db2640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:01.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d78db2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:01.067 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d78db2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:01.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.054+0000 7f8d78db2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:01.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.055+0000 7f8d78db2640 10 monclient(hunting): handle_auth_done global_id 5620 payload 274 2024-09-13T11:57:01.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.055+0000 7f8d78db2640 1 -- 172.21.15.100:0/60911677 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7406ee30 msgr2=0x7f8d7406f250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:57:01.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.055+0000 7f8d78db2640 1 --2- 172.21.15.100:0/60911677 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7406ee30 0x7f8d7406f250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:01.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.055+0000 7f8d78db2640 1 -- 172.21.15.100:0/60911677 >> v1:172.21.15.122:6789/0 conn(0x7f8d7406f790 legacy=0x7f8d741375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:01.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.055+0000 7f8d78db2640 10 monclient: _finish_hunting 0 2024-09-13T11:57:01.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.055+0000 7f8d78db2640 1 monclient: found mon.noname-c 2024-09-13T11:57:01.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.055+0000 7f8d78db2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:01.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.055+0000 7f8d78db2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.122:3300/0 2024-09-13T11:57:01.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.055+0000 7f8d78db2640 1 -- 172.21.15.100:0/60911677 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d7406fca0 con 0x7f8d7406ea50 2024-09-13T11:57:01.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.055+0000 7f8d737fe640 1 -- 172.21.15.100:0/60911677 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1535158299 0 0) 0x7f8d740a1590 con 0x7f8d7406f790 2024-09-13T11:57:01.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.055+0000 7f8d737fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:57:01.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.055+0000 7f8d78db2640 1 --2- 172.21.15.100:0/60911677 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d7406ea50 0x7f8d740a2860 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c0097a0 tx=0x7f8d6c02a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26ca6f20cd626bfb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:01.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.055+0000 7f8d737fe640 1 -- 172.21.15.100:0/60911677 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c037070 con 0x7f8d7406ea50 2024-09-13T11:57:01.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.055+0000 7f8d737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:01.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:57:01.069 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d737fe640 10 monclient: dump: 2024-09-13T11:57:01.069 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:01.069 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:01.069 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:01.069 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:01.069 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:01.069 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:01.069 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:01.069 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:01.070 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:01.070 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:01.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d737fe640 10 monclient: _finish_auth 0 2024-09-13T11:57:01.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d737fe640 10 monclient: _check_auth_tickets 2024-09-13T11:57:01.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:01.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d7b016640 5 monclient: authenticate success, global_id 5620 2024-09-13T11:57:01.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d737fe640 1 -- 172.21.15.100:0/60911677 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d6c02adb0 con 0x7f8d7406ea50 2024-09-13T11:57:01.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d7b016640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:57:01.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d737fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:01.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d7b016640 10 monclient: get_monmap_and_config success 2024-09-13T11:57:01.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d7b016640 4 set_mon_vals no callback set 2024-09-13T11:57:01.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d737fe640 1 -- 172.21.15.100:0/60911677 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c032660 con 0x7f8d7406ea50 2024-09-13T11:57:01.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:01.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:57:01.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d737fe640 10 monclient: dump: 2024-09-13T11:57:01.071 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:01.071 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:01.071 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:01.071 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:01.071 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:01.071 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:01.071 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:01.072 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:01.072 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:01.072 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:01.072 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d7b016640 10 monclient: shutdown 2024-09-13T11:57:01.072 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d7b016640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:01.072 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d7b016640 1 -- 172.21.15.100:0/60911677 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d7406ea50 msgr2=0x7f8d740a2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:01.072 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d7b016640 1 --2- 172.21.15.100:0/60911677 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d7406ea50 0x7f8d740a2860 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c0097a0 tx=0x7f8d6c02a060 comp rx=0 tx=0).stop 2024-09-13T11:57:01.072 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d7b016640 1 -- 172.21.15.100:0/60911677 shutdown_connections 2024-09-13T11:57:01.072 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d7b016640 1 --2- 172.21.15.100:0/60911677 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7406ee30 0x7f8d7406f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:01.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d7b016640 1 --2- 172.21.15.100:0/60911677 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d7406ea50 0x7f8d740a2860 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:01.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d7b016640 1 -- 172.21.15.100:0/60911677 >> 172.21.15.100:0/60911677 conn(0x7f8d740a68e0 msgr2=0x7f8d740a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:01.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d7b016640 1 -- 172.21.15.100:0/60911677 shutdown_connections 2024-09-13T11:57:01.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.056+0000 7f8d7b016640 1 -- 172.21.15.100:0/60911677 wait complete. 2024-09-13T11:57:01.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.057+0000 7f8d7b016640 10 monclient: build_initial_monmap 2024-09-13T11:57:01.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.057+0000 7f8d7b016640 10 monclient: monmap: 2024-09-13T11:57:01.073 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:01.073 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:57:01.073 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:57:01.073 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:57:01.074 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:01.074 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:01.074 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:57:01.074 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:57:01.074 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:57:01.074 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:01.074 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.057+0000 7f8d7b016640 1 Processor -- start 2024-09-13T11:57:01.074 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.057+0000 7f8d7b016640 1 -- start start 2024-09-13T11:57:01.074 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.057+0000 7f8d7b016640 10 monclient: init 2024-09-13T11:57:01.074 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d7b016640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:01.074 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d7b016640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:57:01.074 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d7b016640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7406ea50 0x7f8d741262b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:01.074 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d7b016640 10 monclient(hunting): picked mon.noname-c con 0x7f8d7406ea50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:01.074 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d7b016640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d7406ee30 0x7f8d74123cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:01.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d7b016640 10 monclient(hunting): picked mon.noname-a con 0x7f8d7406ee30 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:57:01.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d7b016640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d7406f790 0x7f8d74124210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:01.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d7b016640 10 monclient(hunting): picked mon.noname-b con 0x7f8d7406f790 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:57:01.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d7b016640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:01.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d7b016640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f8d74137e30 con 0x7f8d7406ee30 2024-09-13T11:57:01.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d7b016640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:01.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d7b016640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f8d74137cb0 con 0x7f8d7406f790 2024-09-13T11:57:01.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d7b016640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:01.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d7b016640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f8d74137fb0 con 0x7f8d7406ea50 2024-09-13T11:57:01.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d7b016640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:01.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d7b016640 10 monclient(hunting): authenticate will time out at 1001.651245s 2024-09-13T11:57:01.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d78db2640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7406ea50 0x7f8d741262b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:01.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d73fff640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d7406ee30 0x7f8d74123cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:01.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d78db2640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7406ea50 0x7f8d741262b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:59092/0 (socket says 172.21.15.100:59092) 2024-09-13T11:57:01.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d78db2640 1 -- 172.21.15.100:0/3899611041 learned_addr learned my addr 172.21.15.100:0/3899611041 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:01.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d73fff640 10 monclient(hunting): get_auth_request con 0x7f8d7406ee30 auth_method 0 2024-09-13T11:57:01.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:01.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d73fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:01.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d73fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:01.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d78db2640 10 monclient(hunting): get_auth_request con 0x7f8d7406ea50 auth_method 0 2024-09-13T11:57:01.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d78db2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:01.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d78db2640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:01.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d78db2640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:01.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:01.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:01.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d795b3640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d7406f790 0x7f8d74124210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:01.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:01.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d795b3640 10 monclient(hunting): get_auth_request con 0x7f8d7406f790 auth_method 0 2024-09-13T11:57:01.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d795b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:01.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d795b3640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:01.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d78db2640 1 -- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7406ea50 msgr2=0x7f8d741262b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-13T11:57:01.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d795b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:01.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d78db2640 1 -- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7406ea50 msgr2=0x7f8d741262b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T11:57:01.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d78db2640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7406ea50 0x7f8d741262b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:57:01.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d78db2640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7406ea50 0x7f8d741262b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:57:01.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d795b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:01.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d795b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:01.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d795b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:01.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d73fff640 10 monclient(hunting): handle_auth_done global_id 14844 payload 824 2024-09-13T11:57:01.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d73fff640 1 -- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7406ea50 msgr2=0x7f8d741262b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:57:01.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d73fff640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d7406ea50 0x7f8d741262b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:01.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d73fff640 1 -- 172.21.15.100:0/3899611041 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d7406f790 msgr2=0x7f8d74124210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:01.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d73fff640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d7406f790 0x7f8d74124210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:01.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d73fff640 10 monclient: _finish_hunting 0 2024-09-13T11:57:01.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d73fff640 1 monclient: found mon.noname-a 2024-09-13T11:57:01.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:01.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d73fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:57:01.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d73fff640 1 -- 172.21.15.100:0/3899611041 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d741277d0 con 0x7f8d7406ee30 2024-09-13T11:57:01.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d73fff640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d7406ee30 0x7f8d74123cd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c00e7e0 tx=0x7f8d5c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:01.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d717fa640 1 -- 172.21.15.100:0/3899611041 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c017070 con 0x7f8d7406ee30 2024-09-13T11:57:01.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:01.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d717fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:57:01.079 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d717fa640 10 monclient: dump: 2024-09-13T11:57:01.079 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:01.079 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:01.079 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:01.079 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:01.079 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:01.079 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:01.079 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:01.079 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:01.079 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:01.079 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:01.079 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d717fa640 10 monclient: _finish_auth 0 2024-09-13T11:57:01.079 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d717fa640 10 monclient: _check_auth_tickets 2024-09-13T11:57:01.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:01.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d7b016640 5 monclient: authenticate success, global_id 14844 2024-09-13T11:57:01.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d717fa640 1 -- 172.21.15.100:0/3899611041 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d5c004020 con 0x7f8d7406ee30 2024-09-13T11:57:01.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d7b016640 10 monclient: _renew_subs 2024-09-13T11:57:01.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d7b016640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:01.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d7b016640 1 -- 172.21.15.100:0/3899611041 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d740a0530 con 0x7f8d7406ee30 2024-09-13T11:57:01.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.058+0000 7f8d717fa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:01.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.059+0000 7f8d717fa640 1 -- 172.21.15.100:0/3899611041 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c00f040 con 0x7f8d7406ee30 2024-09-13T11:57:01.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.059+0000 7f8d717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:01.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.059+0000 7f8d717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:01.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.059+0000 7f8d717fa640 10 monclient: dump: 2024-09-13T11:57:01.080 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:01.080 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:01.081 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:01.081 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:01.081 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:01.081 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:01.081 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:01.081 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:01.081 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:01.081 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:01.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.059+0000 7f8d72ffd640 4 set_mon_vals no callback set 2024-09-13T11:57:01.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.059+0000 7f8d7b016640 10 monclient: _renew_subs 2024-09-13T11:57:01.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.059+0000 7f8d7b016640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:01.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.059+0000 7f8d7b016640 1 -- 172.21.15.100:0/3899611041 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d741274f0 con 0x7f8d7406ee30 2024-09-13T11:57:01.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.059+0000 7f8d717fa640 1 -- 172.21.15.100:0/3899611041 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f8d5c00c6d0 con 0x7f8d7406ee30 2024-09-13T11:57:01.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.059+0000 7f8d7b016640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:01.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.059+0000 7f8d7b016640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d74124750 0x7f8d741cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:01.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.059+0000 7f8d7b016640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:01.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.059+0000 7f8d7b016640 1 -- 172.21.15.100:0/3899611041 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f8d74124b30 con 0x7f8d74124750 2024-09-13T11:57:01.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.060+0000 7f8d78db2640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d74124750 0x7f8d741cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:01.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.060+0000 7f8d717fa640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f8d5403dfa0 0x7f8d54040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:01.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.060+0000 7f8d78db2640 10 monclient: get_auth_request con 0x7f8d74124750 auth_method 0 2024-09-13T11:57:01.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.060+0000 7f8d78db2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:01.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.060+0000 7f8d78db2640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:01.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.060+0000 7f8d78db2640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:01.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.060+0000 7f8d717fa640 1 -- 172.21.15.100:0/3899611041 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(219..219 src has 1..219) ==== 4738+0+0 (secure 0 0 0) 0x7f8d5c052300 con 0x7f8d7406ee30 2024-09-13T11:57:01.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.060+0000 7f8d78db2640 1 -- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d74124750 msgr2=0x7f8d741cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-13T11:57:01.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.060+0000 7f8d78db2640 1 -- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d74124750 msgr2=0x7f8d741cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T11:57:01.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.060+0000 7f8d78db2640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d74124750 0x7f8d741cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:57:01.083 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.060+0000 7f8d78db2640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d74124750 0x7f8d741cc7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:57:01.083 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.060+0000 7f8d795b3640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f8d5403dfa0 0x7f8d54040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:01.083 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.060+0000 7f8d795b3640 10 monclient: get_auth_request con 0x7f8d5403dfa0 auth_method 0 2024-09-13T11:57:01.083 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.060+0000 7f8d795b3640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f8d5403dfa0 0x7f8d54040460 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8d740a8f30 tx=0x7f8d64009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:01.095 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38016]: starting. 2024-09-13T11:57:01.095 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38016]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-38014 2024-09-13T11:57:01.095 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38016]: shutting down. 2024-09-13T11:57:01.263 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.261+0000 7f8d78db2640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d74124750 0x7f8d741cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:01.263 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.261+0000 7f8d78db2640 10 monclient: get_auth_request con 0x7f8d74124750 auth_method 0 2024-09-13T11:57:01.263 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.261+0000 7f8d78db2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:01.263 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.261+0000 7f8d78db2640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:01.264 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.261+0000 7f8d78db2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:57:01.264 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.262+0000 7f8d78db2640 1 -- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d74124750 msgr2=0x7f8d741cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-13T11:57:01.264 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.262+0000 7f8d78db2640 1 -- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d74124750 msgr2=0x7f8d741cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T11:57:01.264 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.262+0000 7f8d78db2640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d74124750 0x7f8d741cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:57:01.264 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.262+0000 7f8d78db2640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d74124750 0x7f8d741cc7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T11:57:01.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.663+0000 7f8d78db2640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d74124750 0x7f8d741cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:01.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.663+0000 7f8d78db2640 10 monclient: get_auth_request con 0x7f8d74124750 auth_method 0 2024-09-13T11:57:01.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.663+0000 7f8d78db2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:01.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.663+0000 7f8d78db2640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:01.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.663+0000 7f8d78db2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:57:01.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.663+0000 7f8d78db2640 1 -- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d74124750 msgr2=0x7f8d741cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-13T11:57:01.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.663+0000 7f8d78db2640 1 -- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d74124750 msgr2=0x7f8d741cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T11:57:01.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.663+0000 7f8d78db2640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d74124750 0x7f8d741cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:57:01.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:01.663+0000 7f8d78db2640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d74124750 0x7f8d741cc7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T11:57:02.466 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.464+0000 7f8d78db2640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d74124750 0x7f8d741cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:02.466 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.464+0000 7f8d78db2640 10 monclient: get_auth_request con 0x7f8d74124750 auth_method 0 2024-09-13T11:57:02.466 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.464+0000 7f8d78db2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:02.466 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.464+0000 7f8d78db2640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:02.466 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.464+0000 7f8d78db2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:57:02.466 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.464+0000 7f8d78db2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:02.466 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.464+0000 7f8d78db2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:02.466 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.464+0000 7f8d78db2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:02.466 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.464+0000 7f8d78db2640 10 monclient(hunting): handle_auth_done global_id 24101 payload 824 2024-09-13T11:57:02.467 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.464+0000 7f8d78db2640 1 -- 172.21.15.100:0/3899611041 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8d741375c0 con 0x7f8d74124750 2024-09-13T11:57:02.467 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.464+0000 7f8d78db2640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d74124750 0x7f8d741cc7d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c000c00 tx=0x7f8d6c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:02.467 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.464+0000 7f8d717fa640 1 -- 172.21.15.100:0/3899611041 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c037040 con 0x7f8d74124750 2024-09-13T11:57:02.467 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.464+0000 7f8d717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:02.467 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.465+0000 7f8d717fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:57:02.467 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.465+0000 7f8d717fa640 10 monclient: dump: 2024-09-13T11:57:02.467 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:02.467 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:02.467 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:02.467 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:02.467 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:02.467 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:02.467 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:02.467 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:02.468 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:02.468 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:02.468 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.465+0000 7f8d717fa640 1 -- 172.21.15.100:0/3899611041 <== mon.2 v2:172.21.15.100:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8d6c004490 con 0x7f8d74124750 2024-09-13T11:57:02.468 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.465+0000 7f8d717fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:02.468 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.465+0000 7f8d717fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:57:02.468 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.465+0000 7f8d717fa640 1 -- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d74124750 msgr2=0x7f8d741cc7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:02.468 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.465+0000 7f8d717fa640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d74124750 0x7f8d741cc7d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c000c00 tx=0x7f8d6c0046c0 comp rx=0 tx=0).stop 2024-09-13T11:57:02.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.465+0000 7f8d717fa640 1 -- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d74124750 msgr2=0x7f8d741cc7d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:57:02.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.465+0000 7f8d717fa640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d74124750 0x7f8d741cc7d0 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c000c00 tx=0x7f8d6c0046c0 comp rx=0 tx=0).stop 2024-09-13T11:57:02.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.472+0000 7f8d7b016640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:57:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.472+0000 7f8d7b016640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d741ccd10 0x7f8d741cd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.472+0000 7f8d7b016640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.472+0000 7f8d7b016640 1 -- 172.21.15.100:0/3899611041 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f8d741cd660 con 0x7f8d741ccd10 2024-09-13T11:57:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.472+0000 7f8d78db2640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d741ccd10 0x7f8d741cd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.472+0000 7f8d78db2640 10 monclient: get_auth_request con 0x7f8d741ccd10 auth_method 0 2024-09-13T11:57:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.472+0000 7f8d78db2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.472+0000 7f8d78db2640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.472+0000 7f8d78db2640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.472+0000 7f8d78db2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.472+0000 7f8d78db2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.472+0000 7f8d78db2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.472+0000 7f8d78db2640 10 monclient(hunting): handle_auth_done global_id 24104 payload 824 2024-09-13T11:57:02.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.472+0000 7f8d78db2640 1 -- 172.21.15.100:0/3899611041 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8d74138370 con 0x7f8d741ccd10 2024-09-13T11:57:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.472+0000 7f8d78db2640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d741ccd10 0x7f8d741cd120 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8d74124cb0 tx=0x7f8d6c009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.472+0000 7f8d717fa640 1 -- 172.21.15.100:0/3899611041 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c04e040 con 0x7f8d741ccd10 2024-09-13T11:57:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.472+0000 7f8d717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.472+0000 7f8d717fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:57:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.472+0000 7f8d717fa640 10 monclient: dump: 2024-09-13T11:57:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:02.475 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:02.476 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:02.476 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:02.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.473+0000 7f8d717fa640 1 -- 172.21.15.100:0/3899611041 <== mon.2 v2:172.21.15.100:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f8d6c040d60 con 0x7f8d741ccd10 2024-09-13T11:57:02.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.473+0000 7f8d717fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:57:02.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.473+0000 7f8d717fa640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:57:02.476 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.473+0000 7f8d717fa640 1 -- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d741ccd10 msgr2=0x7f8d741cd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:02.477 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.473+0000 7f8d717fa640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d741ccd10 0x7f8d741cd120 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8d74124cb0 tx=0x7f8d6c009190 comp rx=0 tx=0).stop 2024-09-13T11:57:02.477 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.473+0000 7f8d717fa640 1 -- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d741ccd10 msgr2=0x7f8d741cd120 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:57:02.477 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.473+0000 7f8d717fa640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d741ccd10 0x7f8d741cd120 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8d74124cb0 tx=0x7f8d6c009190 comp rx=0 tx=0).stop 2024-09-13T11:57:02.477 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:57:02.477 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:57:02.477 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:57:02.477 INFO:teuthology.orchestra.run.smithi100.stdout: "state": "peon", 2024-09-13T11:57:02.477 INFO:teuthology.orchestra.run.smithi100.stdout: "election_epoch": 22, 2024-09-13T11:57:02.477 INFO:teuthology.orchestra.run.smithi100.stdout: "uptime": 15450, 2024-09-13T11:57:02.478 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum": [ 2024-09-13T11:57:02.478 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-13T11:57:02.478 INFO:teuthology.orchestra.run.smithi100.stdout: 1, 2024-09-13T11:57:02.478 INFO:teuthology.orchestra.run.smithi100.stdout: 2 2024-09-13T11:57:02.478 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:02.478 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_age": 4, 2024-09-13T11:57:02.478 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:57:02.478 INFO:teuthology.orchestra.run.smithi100.stdout: "required_con": "2450521708007866372", 2024-09-13T11:57:02.478 INFO:teuthology.orchestra.run.smithi100.stdout: "required_mon": [ 2024-09-13T11:57:02.478 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:57:02.478 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:57:02.478 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:57:02.478 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:57:02.479 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:57:02.479 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:57:02.479 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:57:02.479 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:57:02.479 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:57:02.479 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:57:02.479 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:57:02.479 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:02.479 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:57:02.479 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_mon": [ 2024-09-13T11:57:02.479 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:57:02.479 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:57:02.479 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:57:02.480 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:57:02.480 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:57:02.480 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:57:02.480 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:57:02.480 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:57:02.480 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:57:02.480 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:57:02.480 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:57:02.480 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:02.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:02.480 INFO:teuthology.orchestra.run.smithi100.stdout: "outside_quorum": [], 2024-09-13T11:57:02.480 INFO:teuthology.orchestra.run.smithi100.stdout: "extra_probe_peers": [], 2024-09-13T11:57:02.480 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_provider": [], 2024-09-13T11:57:02.481 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": { 2024-09-13T11:57:02.481 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-09-13T11:57:02.481 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "0b1f4b4d-fad4-4dd3-86ac-a286d80846c0", 2024-09-13T11:57:02.481 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:57:02.481 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:57:02.481 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release": 19, 2024-09-13T11:57:02.481 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release_name": "squid", 2024-09-13T11:57:02.482 INFO:teuthology.orchestra.run.smithi100.stdout: "election_strategy": 1, 2024-09-13T11:57:02.482 INFO:teuthology.orchestra.run.smithi100.stdout: "disallowed_leaders: ": "", 2024-09-13T11:57:02.482 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false, 2024-09-13T11:57:02.482 INFO:teuthology.orchestra.run.smithi100.stdout: "tiebreaker_mon": "", 2024-09-13T11:57:02.482 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_ranks: ": "", 2024-09-13T11:57:02.482 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:57:02.482 INFO:teuthology.orchestra.run.smithi100.stdout: "persistent": [ 2024-09-13T11:57:02.482 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:57:02.483 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:57:02.483 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:57:02.483 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:57:02.483 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:57:02.483 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:57:02.483 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:57:02.483 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:57:02.483 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:57:02.483 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:57:02.483 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:57:02.483 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:02.483 INFO:teuthology.orchestra.run.smithi100.stdout: "optional": [] 2024-09-13T11:57:02.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:02.483 INFO:teuthology.orchestra.run.smithi100.stdout: "mons": [ 2024-09-13T11:57:02.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:02.484 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:57:02.484 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:57:02.484 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:57:02.484 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:57:02.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:02.484 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:57:02.484 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3300", 2024-09-13T11:57:02.484 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:02.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:02.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:02.484 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:57:02.484 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789", 2024-09-13T11:57:02.485 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:02.485 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:02.485 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:02.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:02.485 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789/0", 2024-09-13T11:57:02.485 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-13T11:57:02.485 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:57:02.485 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:57:02.485 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:57:02.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:02.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:02.485 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:57:02.485 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:57:02.486 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:57:02.486 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:57:02.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:02.486 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:57:02.486 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:3300", 2024-09-13T11:57:02.486 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:02.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:02.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:02.486 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:57:02.486 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789", 2024-09-13T11:57:02.486 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:02.486 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:02.487 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:02.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:02.487 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789/0", 2024-09-13T11:57:02.487 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.122:6789/0", 2024-09-13T11:57:02.487 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:57:02.487 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:57:02.487 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:57:02.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:02.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:02.487 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:57:02.487 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:57:02.487 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:57:02.487 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:57:02.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:02.488 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:57:02.488 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3301", 2024-09-13T11:57:02.488 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:02.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:02.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:02.488 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:57:02.488 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790", 2024-09-13T11:57:02.488 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:02.488 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:02.489 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:02.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:02.489 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790/0", 2024-09-13T11:57:02.489 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-13T11:57:02.489 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:57:02.489 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:57:02.489 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:57:02.489 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:02.489 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:02.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:02.489 INFO:teuthology.orchestra.run.smithi100.stdout: "feature_map": { 2024-09-13T11:57:02.489 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": [ 2024-09-13T11:57:02.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:02.490 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:02.490 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:02.490 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:57:02.490 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:02.490 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:02.490 INFO:teuthology.orchestra.run.smithi100.stdout: "client": [ 2024-09-13T11:57:02.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:02.490 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:02.490 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:02.490 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:57:02.490 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:02.490 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:02.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:02.491 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false 2024-09-13T11:57:02.491 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:57:02.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.473+0000 7f8d7b016640 1 -- 172.21.15.100:0/3899611041 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f8d5403dfa0 msgr2=0x7f8d54040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:02.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.473+0000 7f8d7b016640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f8d5403dfa0 0x7f8d54040460 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8d740a8f30 tx=0x7f8d64009040 comp rx=0 tx=0).stop 2024-09-13T11:57:02.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.473+0000 7f8d7b016640 10 monclient: shutdown 2024-09-13T11:57:02.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.473+0000 7f8d7b016640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:02.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.473+0000 7f8d7b016640 1 -- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d7406ee30 msgr2=0x7f8d74123cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:02.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.473+0000 7f8d7b016640 1 --2- 172.21.15.100:0/3899611041 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d7406ee30 0x7f8d74123cd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c00e7e0 tx=0x7f8d5c00eca0 comp rx=0 tx=0).stop 2024-09-13T11:57:02.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.473+0000 7f8d795b3640 1 -- 172.21.15.100:0/3899611041 reap_dead start 2024-09-13T11:57:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.473+0000 7f8d795b3640 1 -- 172.21.15.100:0/3899611041 reap_dead start 2024-09-13T11:57:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.473+0000 7f8d7b016640 1 -- 172.21.15.100:0/3899611041 shutdown_connections 2024-09-13T11:57:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.473+0000 7f8d7b016640 1 -- 172.21.15.100:0/3899611041 >> 172.21.15.100:0/3899611041 conn(0x7f8d740a68e0 msgr2=0x7f8d7409da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.473+0000 7f8d7b016640 1 -- 172.21.15.100:0/3899611041 shutdown_connections 2024-09-13T11:57:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.474+0000 7f8d7b016640 1 -- 172.21.15.100:0/3899611041 wait complete. 2024-09-13T11:57:02.493 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T11:57:02.493 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-13T11:57:02.493 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-13T11:57:02.600 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.599+0000 7f8a3bd89640 10 monclient: get_monmap_and_config 2024-09-13T11:57:02.600 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.599+0000 7f8a3bd89640 10 monclient: build_initial_monmap 2024-09-13T11:57:02.600 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.599+0000 7f8a3bd89640 10 monclient: monmap: 2024-09-13T11:57:02.600 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:02.600 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:02.600 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:57:02.599949+0000 2024-09-13T11:57:02.601 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:57:02.599949+0000 2024-09-13T11:57:02.601 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:02.601 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:02.601 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:57:02.601 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:57:02.601 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:57:02.601 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:57:02.601 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:57:02.601 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:02.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.599+0000 7f8a3bd89640 1 Processor -- start 2024-09-13T11:57:02.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.599+0000 7f8a3bd89640 1 -- start start 2024-09-13T11:57:02.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.599+0000 7f8a3bd89640 10 monclient: init 2024-09-13T11:57:02.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a3bd89640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:02.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a3bd89640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-13T11:57:02.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a3bd89640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a3406ea50 0x7f8a340a2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:02.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a3bd89640 10 monclient(hunting): picked mon.noname-b con 0x7f8a3406ea50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:02.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a3bd89640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8a3406ee30 0x7f8a3406f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:02.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a3bd89640 10 monclient(hunting): picked mon.noname-a con 0x7f8a3406ee30 addr v2:172.21.15.100:3300/0 2024-09-13T11:57:02.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a3bd89640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8a3406f790 addr v1:172.21.15.122:6789/0 2024-09-13T11:57:02.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a3bd89640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a340a1590 con 0x7f8a3406f790 2024-09-13T11:57:02.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a3bd89640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:02.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a3bd89640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f8a340a1870 con 0x7f8a3406ee30 2024-09-13T11:57:02.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a3bd89640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:02.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a3bd89640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f8a340a1b50 con 0x7f8a3406ea50 2024-09-13T11:57:02.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a3bd89640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:02.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a3bd89640 10 monclient(hunting): authenticate will time out at 1003.193431s 2024-09-13T11:57:02.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39b25640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a3406ea50 0x7f8a340a2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:02.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39b25640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a3406ea50 0x7f8a340a2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:46050/0 (socket says 172.21.15.100:46050) 2024-09-13T11:57:02.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39324640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8a3406ee30 0x7f8a3406f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:02.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39b25640 1 -- 172.21.15.100:0/1883296704 learned_addr learned my addr 172.21.15.100:0/1883296704 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:02.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39b25640 10 monclient(hunting): get_auth_request con 0x7f8a3406ea50 auth_method 0 2024-09-13T11:57:02.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39b25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:02.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39b25640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:02.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39b25640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:02.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39324640 10 monclient(hunting): get_auth_request con 0x7f8a3406ee30 auth_method 0 2024-09-13T11:57:02.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39324640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:02.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39324640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:02.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39324640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:02.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39b25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:02.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39b25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:02.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39b25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:02.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39324640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:02.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39324640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:02.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39324640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:02.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39324640 10 monclient(hunting): handle_auth_done global_id 14868 payload 274 2024-09-13T11:57:02.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39324640 1 -- 172.21.15.100:0/1883296704 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a3406ea50 msgr2=0x7f8a340a2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:57:02.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39324640 1 --2- 172.21.15.100:0/1883296704 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a3406ea50 0x7f8a340a2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:02.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39324640 1 -- 172.21.15.100:0/1883296704 >> v1:172.21.15.122:6789/0 conn(0x7f8a3406f790 legacy=0x7f8a341375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:02.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39324640 10 monclient: _finish_hunting 0 2024-09-13T11:57:02.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39324640 1 monclient: found mon.noname-a 2024-09-13T11:57:02.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39324640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:02.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39324640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:57:02.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39324640 1 -- 172.21.15.100:0/1883296704 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a3406fca0 con 0x7f8a3406ee30 2024-09-13T11:57:02.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.600+0000 7f8a39b25640 1 --2- 172.21.15.100:0/1883296704 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a3406ea50 0x7f8a340a2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:57:02.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.601+0000 7f8a39324640 1 --2- 172.21.15.100:0/1883296704 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a3406ee30 0x7f8a3406f250 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8a300097a0 tx=0x7f8a3002a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84ea8435ed742d7e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:02.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.601+0000 7f8a38b23640 1 -- 172.21.15.100:0/1883296704 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a30037070 con 0x7f8a3406ee30 2024-09-13T11:57:02.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.601+0000 7f8a38b23640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:02.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.601+0000 7f8a38b23640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:57:02.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.601+0000 7f8a38b23640 10 monclient: dump: 2024-09-13T11:57:02.614 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:02.614 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:02.614 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:02.614 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:02.614 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:02.614 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:02.614 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:02.614 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:02.614 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:02.614 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:02.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.601+0000 7f8a38b23640 10 monclient: _finish_auth 0 2024-09-13T11:57:02.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.601+0000 7f8a38b23640 10 monclient: _check_auth_tickets 2024-09-13T11:57:02.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.601+0000 7f8a38b23640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:02.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.601+0000 7f8a38b23640 1 -- 172.21.15.100:0/1883296704 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a3002adb0 con 0x7f8a3406ee30 2024-09-13T11:57:02.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.601+0000 7f8a3bd89640 5 monclient: authenticate success, global_id 14868 2024-09-13T11:57:02.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.601+0000 7f8a3bd89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:57:02.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.601+0000 7f8a38b23640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:02.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.601+0000 7f8a3bd89640 10 monclient: get_monmap_and_config success 2024-09-13T11:57:02.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.601+0000 7f8a38b23640 1 -- 172.21.15.100:0/1883296704 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a30032660 con 0x7f8a3406ee30 2024-09-13T11:57:02.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.601+0000 7f8a38b23640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:02.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.601+0000 7f8a3bd89640 4 set_mon_vals no callback set 2024-09-13T11:57:02.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.601+0000 7f8a38b23640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:02.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.601+0000 7f8a38b23640 10 monclient: dump: 2024-09-13T11:57:02.615 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:02.615 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:02.616 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:02.616 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:02.616 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:02.616 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:02.616 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:02.616 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:02.616 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:02.616 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:02.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.602+0000 7f8a3bd89640 10 monclient: shutdown 2024-09-13T11:57:02.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.602+0000 7f8a3bd89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:02.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.602+0000 7f8a3bd89640 1 -- 172.21.15.100:0/1883296704 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a3406ee30 msgr2=0x7f8a3406f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:02.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.602+0000 7f8a3bd89640 1 --2- 172.21.15.100:0/1883296704 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a3406ee30 0x7f8a3406f250 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8a300097a0 tx=0x7f8a3002a060 comp rx=0 tx=0).stop 2024-09-13T11:57:02.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.602+0000 7f8a3bd89640 1 -- 172.21.15.100:0/1883296704 shutdown_connections 2024-09-13T11:57:02.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.602+0000 7f8a3bd89640 1 --2- 172.21.15.100:0/1883296704 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a3406ee30 0x7f8a3406f250 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:02.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.602+0000 7f8a3bd89640 1 --2- 172.21.15.100:0/1883296704 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a3406ea50 0x7f8a340a2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:02.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.602+0000 7f8a3bd89640 1 -- 172.21.15.100:0/1883296704 >> 172.21.15.100:0/1883296704 conn(0x7f8a340a68e0 msgr2=0x7f8a340a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:02.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.602+0000 7f8a3bd89640 1 -- 172.21.15.100:0/1883296704 shutdown_connections 2024-09-13T11:57:02.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.602+0000 7f8a3bd89640 1 -- 172.21.15.100:0/1883296704 wait complete. 2024-09-13T11:57:02.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.602+0000 7f8a3bd89640 10 monclient: build_initial_monmap 2024-09-13T11:57:02.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.602+0000 7f8a3bd89640 10 monclient: monmap: 2024-09-13T11:57:02.617 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:02.617 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:57:02.617 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:57:02.617 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:57:02.617 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:02.617 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:02.617 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:57:02.618 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:57:02.618 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:57:02.618 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:02.618 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3bd89640 1 Processor -- start 2024-09-13T11:57:02.618 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3bd89640 1 -- start start 2024-09-13T11:57:02.618 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3bd89640 10 monclient: init 2024-09-13T11:57:02.618 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3bd89640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:02.618 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3bd89640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:57:02.618 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3bd89640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a3406ea50 0x7f8a341cca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:02.618 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3bd89640 10 monclient(hunting): picked mon.noname-b con 0x7f8a3406ea50 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:57:02.618 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3bd89640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a3406ee30 0x7f8a341ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:02.618 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3bd89640 10 monclient(hunting): picked mon.noname-c con 0x7f8a3406ee30 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:02.618 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3bd89640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a3406f790 0x7f8a34123f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:02.618 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3bd89640 10 monclient(hunting): picked mon.noname-a con 0x7f8a3406f790 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:57:02.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3bd89640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:02.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3bd89640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f8a34137e30 con 0x7f8a3406f790 2024-09-13T11:57:02.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3bd89640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:02.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3bd89640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f8a34137cb0 con 0x7f8a3406ea50 2024-09-13T11:57:02.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3bd89640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:02.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3bd89640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f8a34137fb0 con 0x7f8a3406ee30 2024-09-13T11:57:02.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3bd89640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:02.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3bd89640 10 monclient(hunting): authenticate will time out at 1003.196725s 2024-09-13T11:57:02.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a39324640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a3406ee30 0x7f8a341ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:02.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3a326640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a3406f790 0x7f8a34123f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:02.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a39324640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a3406ee30 0x7f8a341ccf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:46064/0 (socket says 172.21.15.100:46064) 2024-09-13T11:57:02.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a39324640 1 -- 172.21.15.100:0/1032261659 learned_addr learned my addr 172.21.15.100:0/1032261659 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:02.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a39324640 10 monclient(hunting): get_auth_request con 0x7f8a3406ee30 auth_method 0 2024-09-13T11:57:02.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a39324640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:02.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a39324640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:02.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a39324640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:02.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3a326640 10 monclient(hunting): get_auth_request con 0x7f8a3406f790 auth_method 0 2024-09-13T11:57:02.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3a326640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:02.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3a326640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:02.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3a326640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:02.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a39b25640 1 --2- 172.21.15.100:0/1032261659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a3406ea50 0x7f8a341cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:02.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a39b25640 10 monclient(hunting): get_auth_request con 0x7f8a3406ea50 auth_method 0 2024-09-13T11:57:02.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a39b25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:02.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a39b25640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:02.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a39b25640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:02.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a39324640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:02.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a39324640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:02.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a39324640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:02.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3a326640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:02.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3a326640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:02.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.603+0000 7f8a3a326640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:02.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a39b25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:02.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a39b25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:02.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a39b25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:02.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a39324640 10 monclient(hunting): handle_auth_done global_id 24110 payload 824 2024-09-13T11:57:02.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a39324640 1 -- 172.21.15.100:0/1032261659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a3406ea50 msgr2=0x7f8a341cca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:02.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a39324640 1 --2- 172.21.15.100:0/1032261659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a3406ea50 0x7f8a341cca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:02.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a39324640 1 -- 172.21.15.100:0/1032261659 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a3406f790 msgr2=0x7f8a34123f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:02.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a39324640 1 --2- 172.21.15.100:0/1032261659 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a3406f790 0x7f8a34123f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:02.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a39324640 10 monclient: _finish_hunting 0 2024-09-13T11:57:02.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a39324640 1 monclient: found mon.noname-c 2024-09-13T11:57:02.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a39324640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:02.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a39324640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:57:02.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a39324640 1 -- 172.21.15.100:0/1032261659 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a34127970 con 0x7f8a3406ee30 2024-09-13T11:57:02.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a3a326640 1 --2- 172.21.15.100:0/1032261659 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a3406f790 0x7f8a34123f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:57:02.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a39324640 1 --2- 172.21.15.100:0/1032261659 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a3406ee30 0x7f8a341ccf80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8a30035040 tx=0x7f8a300046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:02.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a2a7fc640 1 -- 172.21.15.100:0/1032261659 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a30041070 con 0x7f8a3406ee30 2024-09-13T11:57:02.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a2a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:02.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a2a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:57:02.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a2a7fc640 10 monclient: dump: 2024-09-13T11:57:02.622 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:02.622 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:02.622 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:02.622 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:02.623 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:02.623 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:02.623 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:02.623 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:02.623 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:02.623 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:02.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a2a7fc640 10 monclient: _finish_auth 0 2024-09-13T11:57:02.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a2a7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:57:02.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:02.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a2a7fc640 1 -- 172.21.15.100:0/1032261659 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a3003c680 con 0x7f8a3406ee30 2024-09-13T11:57:02.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a3bd89640 5 monclient: authenticate success, global_id 24110 2024-09-13T11:57:02.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a3bd89640 10 monclient: _renew_subs 2024-09-13T11:57:02.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a3bd89640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:57:02.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a3bd89640 1 -- 172.21.15.100:0/1032261659 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a34127c50 con 0x7f8a3406ee30 2024-09-13T11:57:02.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a2a7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:02.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a2a7fc640 1 -- 172.21.15.100:0/1032261659 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a30037040 con 0x7f8a3406ee30 2024-09-13T11:57:02.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a2a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:02.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a2a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:57:02.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a2a7fc640 10 monclient: dump: 2024-09-13T11:57:02.624 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:02.624 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:02.624 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:02.624 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:02.624 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:02.624 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:02.624 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:02.624 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:02.625 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:02.625 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:02.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a3bd89640 10 monclient: _renew_subs 2024-09-13T11:57:02.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a3bd89640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:57:02.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a3bd89640 1 -- 172.21.15.100:0/1032261659 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a34137210 con 0x7f8a3406ee30 2024-09-13T11:57:02.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.604+0000 7f8a2bfff640 4 set_mon_vals no callback set 2024-09-13T11:57:02.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.605+0000 7f8a3bd89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:02.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.605+0000 7f8a3bd89640 1 --2- 172.21.15.100:0/1032261659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a34124460 0x7f8a34124840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:02.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.605+0000 7f8a3bd89640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:02.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.605+0000 7f8a3bd89640 1 -- 172.21.15.100:0/1032261659 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f8a34127560 con 0x7f8a34124460 2024-09-13T11:57:02.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.605+0000 7f8a2a7fc640 1 -- 172.21.15.100:0/1032261659 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f8a3003a070 con 0x7f8a3406ee30 2024-09-13T11:57:02.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.605+0000 7f8a39b25640 1 --2- 172.21.15.100:0/1032261659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a34124460 0x7f8a34124840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:02.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.605+0000 7f8a39b25640 10 monclient: get_auth_request con 0x7f8a34124460 auth_method 0 2024-09-13T11:57:02.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.605+0000 7f8a39b25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:02.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.605+0000 7f8a39b25640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:02.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.605+0000 7f8a39b25640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:02.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.605+0000 7f8a2a7fc640 1 --2- 172.21.15.100:0/1032261659 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f8a1803dfa0 0x7f8a18040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:02.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.605+0000 7f8a39b25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:02.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.605+0000 7f8a39b25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:02.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.605+0000 7f8a39b25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:02.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.605+0000 7f8a2a7fc640 1 -- 172.21.15.100:0/1032261659 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(221..221 src has 1..221) ==== 4738+0+0 (secure 0 0 0) 0x7f8a30072a80 con 0x7f8a3406ee30 2024-09-13T11:57:02.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.605+0000 7f8a39b25640 10 monclient(hunting): handle_auth_done global_id 5653 payload 824 2024-09-13T11:57:02.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.605+0000 7f8a3a326640 1 --2- 172.21.15.100:0/1032261659 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f8a1803dfa0 0x7f8a18040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:02.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.605+0000 7f8a39b25640 1 -- 172.21.15.100:0/1032261659 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8a340a2860 con 0x7f8a34124460 2024-09-13T11:57:02.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.605+0000 7f8a3a326640 10 monclient: get_auth_request con 0x7f8a1803dfa0 auth_method 0 2024-09-13T11:57:02.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.606+0000 7f8a39b25640 1 --2- 172.21.15.100:0/1032261659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a34124460 0x7f8a34124840 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f8a24006ca0 tx=0x7f8a24011d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:02.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.606+0000 7f8a2a7fc640 1 -- 172.21.15.100:0/1032261659 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2400faa0 con 0x7f8a34124460 2024-09-13T11:57:02.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.606+0000 7f8a2a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:02.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.606+0000 7f8a2a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:57:02.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.606+0000 7f8a2a7fc640 10 monclient: dump: 2024-09-13T11:57:02.627 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:02.627 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:02.627 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:02.627 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:02.627 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:02.627 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:02.627 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:02.627 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:02.627 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:02.627 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:02.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.606+0000 7f8a3a326640 1 --2- 172.21.15.100:0/1032261659 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f8a1803dfa0 0x7f8a18040460 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c0012e0 tx=0x7f8a2c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:02.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.606+0000 7f8a2a7fc640 1 -- 172.21.15.100:0/1032261659 <== mon.1 v2:172.21.15.122:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8a2400faa0 con 0x7f8a34124460 2024-09-13T11:57:02.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.606+0000 7f8a2a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:02.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.606+0000 7f8a2a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:57:02.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.606+0000 7f8a2a7fc640 1 -- 172.21.15.100:0/1032261659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a34124460 msgr2=0x7f8a34124840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:02.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.606+0000 7f8a2a7fc640 1 --2- 172.21.15.100:0/1032261659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a34124460 0x7f8a34124840 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f8a24006ca0 tx=0x7f8a24011d80 comp rx=0 tx=0).stop 2024-09-13T11:57:02.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.606+0000 7f8a2a7fc640 1 -- 172.21.15.100:0/1032261659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a34124460 msgr2=0x7f8a34124840 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:57:02.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.606+0000 7f8a2a7fc640 1 --2- 172.21.15.100:0/1032261659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a34124460 0x7f8a34124840 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:02.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.612+0000 7f8a3bd89640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:57:02.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.612+0000 7f8a3bd89640 1 --2- 172.21.15.100:0/1032261659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a34124d80 0x7f8a34120430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:02.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.612+0000 7f8a3bd89640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:02.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.612+0000 7f8a3bd89640 1 -- 172.21.15.100:0/1032261659 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f8a34069100 con 0x7f8a34124d80 2024-09-13T11:57:02.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.612+0000 7f8a39b25640 1 --2- 172.21.15.100:0/1032261659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a34124d80 0x7f8a34120430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:02.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.612+0000 7f8a39b25640 10 monclient: get_auth_request con 0x7f8a34124d80 auth_method 0 2024-09-13T11:57:02.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.612+0000 7f8a39b25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:02.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.612+0000 7f8a39b25640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:02.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.612+0000 7f8a39b25640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:02.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.612+0000 7f8a39b25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:02.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.612+0000 7f8a39b25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:02.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.612+0000 7f8a39b25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:02.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.613+0000 7f8a39b25640 10 monclient(hunting): handle_auth_done global_id 5656 payload 824 2024-09-13T11:57:02.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.613+0000 7f8a39b25640 1 -- 172.21.15.100:0/1032261659 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8a34126450 con 0x7f8a34124d80 2024-09-13T11:57:02.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.613+0000 7f8a39b25640 1 --2- 172.21.15.100:0/1032261659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a34124d80 0x7f8a34120430 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8a24006ca0 tx=0x7f8a240046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:02.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.613+0000 7f8a2a7fc640 1 -- 172.21.15.100:0/1032261659 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a24014870 con 0x7f8a34124d80 2024-09-13T11:57:02.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.613+0000 7f8a2a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:02.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.613+0000 7f8a2a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:57:02.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.613+0000 7f8a2a7fc640 10 monclient: dump: 2024-09-13T11:57:02.631 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:02.631 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:02.632 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:02.632 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:02.632 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:02.632 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:02.632 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:02.632 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:02.632 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:02.632 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:02.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.613+0000 7f8a2a7fc640 1 -- 172.21.15.100:0/1032261659 <== mon.1 v2:172.21.15.122:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f8a2402c400 con 0x7f8a34124d80 2024-09-13T11:57:02.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.613+0000 7f8a2a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:57:02.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.613+0000 7f8a2a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:57:02.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.613+0000 7f8a2a7fc640 1 -- 172.21.15.100:0/1032261659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a34124d80 msgr2=0x7f8a34120430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:02.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.613+0000 7f8a2a7fc640 1 --2- 172.21.15.100:0/1032261659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a34124d80 0x7f8a34120430 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8a24006ca0 tx=0x7f8a240046c0 comp rx=0 tx=0).stop 2024-09-13T11:57:02.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.613+0000 7f8a2a7fc640 1 -- 172.21.15.100:0/1032261659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a34124d80 msgr2=0x7f8a34120430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:57:02.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.613+0000 7f8a2a7fc640 1 --2- 172.21.15.100:0/1032261659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a34124d80 0x7f8a34120430 secure :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8a24006ca0 tx=0x7f8a240046c0 comp rx=0 tx=0).stop 2024-09-13T11:57:02.633 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:57:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:57:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:57:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "state": "peon", 2024-09-13T11:57:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "election_epoch": 22, 2024-09-13T11:57:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "uptime": 261643, 2024-09-13T11:57:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum": [ 2024-09-13T11:57:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-13T11:57:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: 1, 2024-09-13T11:57:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: 2 2024-09-13T11:57:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_age": 4, 2024-09-13T11:57:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:57:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "required_con": "2450521708007866372", 2024-09-13T11:57:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "required_mon": [ 2024-09-13T11:57:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:57:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:57:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:57:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:57:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:57:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:57:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:57:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:57:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:57:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:57:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:57:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:57:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_mon": [ 2024-09-13T11:57:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:57:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:57:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:57:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:57:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:57:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:57:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:57:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:57:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:57:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:57:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:57:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "outside_quorum": [], 2024-09-13T11:57:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "extra_probe_peers": [], 2024-09-13T11:57:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_provider": [], 2024-09-13T11:57:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": { 2024-09-13T11:57:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-09-13T11:57:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "0b1f4b4d-fad4-4dd3-86ac-a286d80846c0", 2024-09-13T11:57:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:57:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:57:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release": 19, 2024-09-13T11:57:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release_name": "squid", 2024-09-13T11:57:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "election_strategy": 1, 2024-09-13T11:57:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "disallowed_leaders: ": "", 2024-09-13T11:57:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false, 2024-09-13T11:57:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "tiebreaker_mon": "", 2024-09-13T11:57:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_ranks: ": "", 2024-09-13T11:57:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:57:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "persistent": [ 2024-09-13T11:57:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:57:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:57:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:57:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:57:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:57:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:57:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:57:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:57:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:57:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:57:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:57:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "optional": [] 2024-09-13T11:57:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "mons": [ 2024-09-13T11:57:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:57:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:57:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:57:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:57:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:57:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3300", 2024-09-13T11:57:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:57:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789", 2024-09-13T11:57:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789/0", 2024-09-13T11:57:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-13T11:57:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:57:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:57:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:57:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:57:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:57:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:57:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:57:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:57:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:3300", 2024-09-13T11:57:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:57:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789", 2024-09-13T11:57:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789/0", 2024-09-13T11:57:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.122:6789/0", 2024-09-13T11:57:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:57:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:57:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:57:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:57:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:57:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:57:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:57:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:57:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3301", 2024-09-13T11:57:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:57:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790", 2024-09-13T11:57:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790/0", 2024-09-13T11:57:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-13T11:57:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:57:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:57:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:57:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "feature_map": { 2024-09-13T11:57:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": [ 2024-09-13T11:57:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:57:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": [ 2024-09-13T11:57:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 4 2024-09-13T11:57:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "client": [ 2024-09-13T11:57:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 6 2024-09-13T11:57:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": [ 2024-09-13T11:57:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:57:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false 2024-09-13T11:57:02.646 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:57:02.648 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.614+0000 7f8a3bd89640 1 -- 172.21.15.100:0/1032261659 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f8a1803dfa0 msgr2=0x7f8a18040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:02.648 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.614+0000 7f8a3bd89640 1 --2- 172.21.15.100:0/1032261659 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f8a1803dfa0 0x7f8a18040460 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c0012e0 tx=0x7f8a2c009040 comp rx=0 tx=0).stop 2024-09-13T11:57:02.648 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.614+0000 7f8a3bd89640 10 monclient: shutdown 2024-09-13T11:57:02.648 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.614+0000 7f8a3bd89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:02.648 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.614+0000 7f8a3bd89640 1 -- 172.21.15.100:0/1032261659 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a3406ee30 msgr2=0x7f8a341ccf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:02.648 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.614+0000 7f8a3bd89640 1 --2- 172.21.15.100:0/1032261659 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a3406ee30 0x7f8a341ccf80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8a30035040 tx=0x7f8a300046c0 comp rx=0 tx=0).stop 2024-09-13T11:57:02.648 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.614+0000 7f8a3a326640 1 -- 172.21.15.100:0/1032261659 reap_dead start 2024-09-13T11:57:02.648 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.614+0000 7f8a3a326640 1 -- 172.21.15.100:0/1032261659 reap_dead start 2024-09-13T11:57:02.648 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.614+0000 7f8a3bd89640 1 -- 172.21.15.100:0/1032261659 shutdown_connections 2024-09-13T11:57:02.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.614+0000 7f8a3bd89640 1 -- 172.21.15.100:0/1032261659 >> 172.21.15.100:0/1032261659 conn(0x7f8a340a68e0 msgr2=0x7f8a3409fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:02.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.614+0000 7f8a3bd89640 1 -- 172.21.15.100:0/1032261659 shutdown_connections 2024-09-13T11:57:02.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.614+0000 7f8a3bd89640 1 -- 172.21.15.100:0/1032261659 wait complete. 2024-09-13T11:57:02.649 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T11:57:02.649 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-13T11:57:02.649 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-13T11:57:02.746 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.745+0000 7f9e074b6640 10 monclient: get_monmap_and_config 2024-09-13T11:57:02.746 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.745+0000 7f9e074b6640 10 monclient: build_initial_monmap 2024-09-13T11:57:02.746 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.745+0000 7f9e074b6640 10 monclient: monmap: 2024-09-13T11:57:02.746 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:02.746 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:02.746 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:57:02.745964+0000 2024-09-13T11:57:02.746 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:57:02.745964+0000 2024-09-13T11:57:02.746 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:02.747 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:02.747 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:57:02.747 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:57:02.747 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:57:02.747 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:57:02.747 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:57:02.747 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:02.754 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.745+0000 7f9e074b6640 1 Processor -- start 2024-09-13T11:57:02.754 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e074b6640 1 -- start start 2024-09-13T11:57:02.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e074b6640 10 monclient: init 2024-09-13T11:57:02.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e074b6640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:02.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e074b6640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-13T11:57:02.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e074b6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e0006ea50 addr v1:172.21.15.100:6789/0 2024-09-13T11:57:02.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e074b6640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e000a2860 0x7f9e0006f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:02.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e074b6640 10 monclient(hunting): picked mon.noname-b con 0x7f9e000a2860 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:02.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e074b6640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9e0013c9f0 0x7f9e0013ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:02.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e074b6640 10 monclient(hunting): picked mon.noname-c con 0x7f9e0013c9f0 addr v2:172.21.15.122:3300/0 2024-09-13T11:57:02.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e074b6640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e0006fa80 con 0x7f9e0006ea50 2024-09-13T11:57:02.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e074b6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:02.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e074b6640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f9e000a2c80 con 0x7f9e0013c9f0 2024-09-13T11:57:02.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e074b6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:02.755 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e074b6640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f9e000a15e0 con 0x7f9e000a2860 2024-09-13T11:57:02.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e074b6640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:02.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e074b6640 10 monclient(hunting): authenticate will time out at 1003.339632s 2024-09-13T11:57:02.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e05252640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f9e0006ea50 0x7f9e0006ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:47462/0 (socket says 172.21.15.100:47462) 2024-09-13T11:57:02.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e05252640 1 -- 172.21.15.100:0/2733315462 learned_addr learned my addr 172.21.15.100:0/2733315462 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:57:02.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e04a51640 1 --2- 172.21.15.100:0/2733315462 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e000a2860 0x7f9e0006f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:02.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e04a51640 10 monclient(hunting): get_auth_request con 0x7f9e000a2860 auth_method 0 2024-09-13T11:57:02.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e04a51640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:02.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e04a51640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:02.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e04a51640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:02.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e04a51640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:02.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e04a51640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:02.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e04a51640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:02.756 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9df7fff640 1 -- 172.21.15.100:0/2733315462 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4120373159 0 0) 0x7f9dfc001110 con 0x7f9e0006ea50 2024-09-13T11:57:02.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9df7fff640 10 monclient(hunting): _negotiate 2024-09-13T11:57:02.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9df7fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:02.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9df7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:02.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9df7fff640 10 monclient(hunting): my global_id is 14874 2024-09-13T11:57:02.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9df7fff640 1 -- 172.21.15.100:0/2733315462 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9dec003520 con 0x7f9e0006ea50 2024-09-13T11:57:02.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e04a51640 10 monclient(hunting): handle_auth_done global_id 24113 payload 274 2024-09-13T11:57:02.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e04a51640 1 -- 172.21.15.100:0/2733315462 >> v2:172.21.15.122:3300/0 conn(0x7f9e0013c9f0 msgr2=0x7f9e0013ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:57:02.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e04a51640 1 --2- 172.21.15.100:0/2733315462 >> v2:172.21.15.122:3300/0 conn(0x7f9e0013c9f0 0x7f9e0013ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:02.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e04a51640 1 -- 172.21.15.100:0/2733315462 >> v1:172.21.15.100:6789/0 conn(0x7f9e0006ea50 legacy=0x7f9e0006ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:02.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e04a51640 10 monclient: _finish_hunting 0 2024-09-13T11:57:02.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e04a51640 1 monclient: found mon.noname-b 2024-09-13T11:57:02.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e04a51640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:02.757 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e04a51640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:57:02.758 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.746+0000 7f9e04a51640 1 -- 172.21.15.100:0/2733315462 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e000a1870 con 0x7f9e000a2860 2024-09-13T11:57:02.758 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.747+0000 7f9e04a51640 1 --2- 172.21.15.100:0/2733315462 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e000a2860 0x7f9e0006f540 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9df00097a0 tx=0x7f9df002a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b6d12cb9e26e54a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:02.758 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.747+0000 7f9df7fff640 1 -- 172.21.15.100:0/2733315462 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df0037070 con 0x7f9e000a2860 2024-09-13T11:57:02.758 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.747+0000 7f9df7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:02.758 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.747+0000 7f9df7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:57:02.758 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.747+0000 7f9df7fff640 10 monclient: dump: 2024-09-13T11:57:02.758 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:02.758 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:02.758 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:02.758 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:02.758 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:02.758 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:02.758 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:02.758 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:02.759 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:02.759 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:02.759 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.747+0000 7f9df7fff640 10 monclient: _finish_auth 0 2024-09-13T11:57:02.759 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.747+0000 7f9df7fff640 10 monclient: _check_auth_tickets 2024-09-13T11:57:02.759 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.747+0000 7f9df7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:02.759 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.747+0000 7f9e074b6640 5 monclient: authenticate success, global_id 24113 2024-09-13T11:57:02.759 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.747+0000 7f9df7fff640 1 -- 172.21.15.100:0/2733315462 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9df002adb0 con 0x7f9e000a2860 2024-09-13T11:57:02.759 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.747+0000 7f9e074b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:57:02.759 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.747+0000 7f9df7fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:02.759 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.747+0000 7f9e074b6640 10 monclient: get_monmap_and_config success 2024-09-13T11:57:02.759 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.747+0000 7f9e074b6640 4 set_mon_vals no callback set 2024-09-13T11:57:02.759 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.747+0000 7f9df7fff640 1 -- 172.21.15.100:0/2733315462 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df0032660 con 0x7f9e000a2860 2024-09-13T11:57:02.759 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.747+0000 7f9df7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:02.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.747+0000 7f9df7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:57:02.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.747+0000 7f9df7fff640 10 monclient: dump: 2024-09-13T11:57:02.760 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:02.760 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:02.760 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:02.760 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:02.760 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:02.760 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:02.760 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:02.760 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:02.760 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:02.760 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:02.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.748+0000 7f9e074b6640 10 monclient: shutdown 2024-09-13T11:57:02.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.748+0000 7f9e074b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:02.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.748+0000 7f9e074b6640 1 -- 172.21.15.100:0/2733315462 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e000a2860 msgr2=0x7f9e0006f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:02.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.748+0000 7f9e074b6640 1 --2- 172.21.15.100:0/2733315462 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e000a2860 0x7f9e0006f540 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9df00097a0 tx=0x7f9df002a060 comp rx=0 tx=0).stop 2024-09-13T11:57:02.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.748+0000 7f9e074b6640 1 -- 172.21.15.100:0/2733315462 shutdown_connections 2024-09-13T11:57:02.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.748+0000 7f9e074b6640 1 --2- 172.21.15.100:0/2733315462 >> v2:172.21.15.122:3300/0 conn(0x7f9e0013c9f0 0x7f9e0013ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:02.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.748+0000 7f9e074b6640 1 --2- 172.21.15.100:0/2733315462 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e000a2860 0x7f9e0006f540 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:02.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.748+0000 7f9e074b6640 1 -- 172.21.15.100:0/2733315462 >> 172.21.15.100:0/2733315462 conn(0x7f9e000a68e0 msgr2=0x7f9e000a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:02.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.748+0000 7f9e074b6640 1 -- 172.21.15.100:0/2733315462 shutdown_connections 2024-09-13T11:57:02.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.748+0000 7f9e074b6640 1 -- 172.21.15.100:0/2733315462 wait complete. 2024-09-13T11:57:02.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.748+0000 7f9e074b6640 10 monclient: build_initial_monmap 2024-09-13T11:57:02.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.748+0000 7f9e074b6640 10 monclient: monmap: 2024-09-13T11:57:02.761 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:02.761 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:57:02.761 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:57:02.762 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:57:02.762 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:02.762 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:02.762 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:57:02.762 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:57:02.762 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:57:02.762 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:02.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e074b6640 1 Processor -- start 2024-09-13T11:57:02.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e074b6640 1 -- start start 2024-09-13T11:57:02.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e074b6640 10 monclient: init 2024-09-13T11:57:02.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e074b6640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:02.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e074b6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:57:02.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e074b6640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e0006ea50 0x7f9e001cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:02.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e074b6640 10 monclient(hunting): picked mon.noname-c con 0x7f9e0006ea50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:02.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e074b6640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e000a2860 0x7f9e00137900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:02.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e074b6640 10 monclient(hunting): picked mon.noname-a con 0x7f9e000a2860 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:57:02.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e074b6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e0013c9f0 0x7f9e0009eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:02.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e074b6640 10 monclient(hunting): picked mon.noname-b con 0x7f9e0013c9f0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:57:02.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e074b6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:02.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e074b6640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f9e000a5be0 con 0x7f9e000a2860 2024-09-13T11:57:02.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e074b6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:02.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e074b6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9e000a5a60 con 0x7f9e0013c9f0 2024-09-13T11:57:02.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e074b6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:02.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e074b6640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f9e0013f320 con 0x7f9e0006ea50 2024-09-13T11:57:02.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e074b6640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:02.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e074b6640 10 monclient(hunting): authenticate will time out at 1003.342890s 2024-09-13T11:57:02.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e05252640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e0006ea50 0x7f9e001cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:02.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e04a51640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e000a2860 0x7f9e00137900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:02.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e04a51640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e000a2860 0x7f9e00137900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55376/0 (socket says 172.21.15.100:55376) 2024-09-13T11:57:02.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e04a51640 1 -- 172.21.15.100:0/2176512379 learned_addr learned my addr 172.21.15.100:0/2176512379 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:02.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e04a51640 10 monclient(hunting): get_auth_request con 0x7f9e000a2860 auth_method 0 2024-09-13T11:57:02.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.749+0000 7f9e05252640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e0006ea50 0x7f9e001cc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:46080/0 (socket says 172.21.15.100:46080) 2024-09-13T11:57:02.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e04a51640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:02.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e04a51640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:02.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e04a51640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:02.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e05252640 10 monclient(hunting): get_auth_request con 0x7f9e0006ea50 auth_method 0 2024-09-13T11:57:02.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e05252640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:02.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e05252640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:02.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e05252640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:02.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e04a51640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:02.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e04a51640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:02.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e04a51640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:02.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e05252640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:02.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e05252640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:02.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e05252640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:02.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e05a53640 1 --2- 172.21.15.100:0/2176512379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e0013c9f0 0x7f9e0009eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:02.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e05a53640 10 monclient(hunting): get_auth_request con 0x7f9e0013c9f0 auth_method 0 2024-09-13T11:57:02.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e05a53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:02.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e05a53640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:02.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e05a53640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:02.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e04a51640 10 monclient(hunting): handle_auth_done global_id 14877 payload 824 2024-09-13T11:57:02.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e04a51640 1 -- 172.21.15.100:0/2176512379 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e0006ea50 msgr2=0x7f9e001cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:02.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e04a51640 1 --2- 172.21.15.100:0/2176512379 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e0006ea50 0x7f9e001cc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:02.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e04a51640 1 -- 172.21.15.100:0/2176512379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e0013c9f0 msgr2=0x7f9e0009eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:02.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e04a51640 1 --2- 172.21.15.100:0/2176512379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e0013c9f0 0x7f9e0009eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:02.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e04a51640 10 monclient: _finish_hunting 0 2024-09-13T11:57:02.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e04a51640 1 monclient: found mon.noname-a 2024-09-13T11:57:02.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e04a51640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:02.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e04a51640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:57:02.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e04a51640 1 -- 172.21.15.100:0/2176512379 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e00122290 con 0x7f9e000a2860 2024-09-13T11:57:02.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e05252640 1 --2- 172.21.15.100:0/2176512379 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e0006ea50 0x7f9e001cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:57:02.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e04a51640 1 --2- 172.21.15.100:0/2176512379 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e000a2860 0x7f9e00137900 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9df0000c00 tx=0x7f9df00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:02.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9df5ffb640 1 -- 172.21.15.100:0/2176512379 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df0042070 con 0x7f9e000a2860 2024-09-13T11:57:02.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9df5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:02.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9df5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:57:02.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9df5ffb640 10 monclient: dump: 2024-09-13T11:57:02.766 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:02.767 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:02.767 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:02.767 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:02.767 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:02.767 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:02.767 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:02.767 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:02.767 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:02.767 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:02.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9df5ffb640 10 monclient: _finish_auth 0 2024-09-13T11:57:02.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9df5ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:57:02.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9df5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:02.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9df5ffb640 1 -- 172.21.15.100:0/2176512379 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9df003e460 con 0x7f9e000a2860 2024-09-13T11:57:02.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9df5ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:02.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9df5ffb640 1 -- 172.21.15.100:0/2176512379 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df0037040 con 0x7f9e000a2860 2024-09-13T11:57:02.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9df5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:02.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9df5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:02.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9df5ffb640 10 monclient: dump: 2024-09-13T11:57:02.768 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:02.768 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:02.768 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:02.768 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:02.768 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:02.768 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:02.768 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:02.768 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:02.768 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:02.769 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:02.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e074b6640 5 monclient: authenticate success, global_id 14877 2024-09-13T11:57:02.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e074b6640 10 monclient: _renew_subs 2024-09-13T11:57:02.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e074b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:02.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e074b6640 1 -- 172.21.15.100:0/2176512379 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e0013f090 con 0x7f9e000a2860 2024-09-13T11:57:02.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9df77fe640 4 set_mon_vals no callback set 2024-09-13T11:57:02.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e074b6640 10 monclient: _renew_subs 2024-09-13T11:57:02.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e074b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:02.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.750+0000 7f9e074b6640 1 -- 172.21.15.100:0/2176512379 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e00121fb0 con 0x7f9e000a2860 2024-09-13T11:57:02.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.751+0000 7f9df5ffb640 1 -- 172.21.15.100:0/2176512379 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f9df0045070 con 0x7f9e000a2860 2024-09-13T11:57:02.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.751+0000 7f9e074b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:02.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.751+0000 7f9e074b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:02.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.751+0000 7f9e074b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:02.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.751+0000 7f9e074b6640 1 -- 172.21.15.100:0/2176512379 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e0006ee30 con 0x7f9e000a2860 2024-09-13T11:57:02.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.751+0000 7f9df5ffb640 1 --2- 172.21.15.100:0/2176512379 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9de403dfa0 0x7f9de4040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:02.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.751+0000 7f9df5ffb640 1 -- 172.21.15.100:0/2176512379 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(221..221 src has 1..221) ==== 4738+0+0 (secure 0 0 0) 0x7f9df0073010 con 0x7f9e000a2860 2024-09-13T11:57:02.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.752+0000 7f9e05252640 1 --2- 172.21.15.100:0/2176512379 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9de403dfa0 0x7f9de4040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:02.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.752+0000 7f9e05252640 10 monclient: get_auth_request con 0x7f9de403dfa0 auth_method 0 2024-09-13T11:57:02.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.752+0000 7f9e05252640 1 --2- 172.21.15.100:0/2176512379 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9de403dfa0 0x7f9de4040460 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc005880 tx=0x7f9dfc009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:02.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.756+0000 7f9df5ffb640 1 -- 172.21.15.100:0/2176512379 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f9df0045d70 con 0x7f9e000a2860 2024-09-13T11:57:02.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.756+0000 7f9df5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:02.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.756+0000 7f9df5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:57:02.997 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.996+0000 7f9e074b6640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-13T11:57:02.997 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.996+0000 7f9e074b6640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-13T11:57:02.997 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.996+0000 7f9e074b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:02.997 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.996+0000 7f9e074b6640 1 -- 172.21.15.100:0/2176512379 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f9e00069100 con 0x7f9e000a2860 2024-09-13T11:57:02.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.996+0000 7f9df5ffb640 1 -- 172.21.15.100:0/2176512379 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f9df003c430 con 0x7f9e000a2860 2024-09-13T11:57:02.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.997+0000 7f9df5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-13T11:57:02.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.997+0000 7f9df5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:57:02.999 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:02.997+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:02.999 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:02.997+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=1,health=12,kv=13,logm=66} crc {auth=2392644224,config=1228700967,health=2157093694,kv=4190740400,logm=866273049}) 2024-09-13T11:57:02.999 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:02.997+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=11,kv=12,logm=66} crc {auth=471287096,config=1097430493,health=1886805112,kv=296939423,logm=2467255179}) 2024-09-13T11:57:02.999 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:02.997+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:02.999 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:02.997+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=1,health=12,kv=13,logm=66} crc {auth=2392644224,config=1228700967,health=2157093694,kv=4190740400,logm=866273049}) 2024-09-13T11:57:02.999 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:02.997+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=9,config=2,health=12,kv=13,logm=64} crc {auth=471287096,config=1097430493,health=2157093694,kv=4190740400,logm=1332123473}) 2024-09-13T11:57:03.000 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:02.998+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.000 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:02.998+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=631795057}) 2024-09-13T11:57:03.000 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:02.998+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1661570857}) 2024-09-13T11:57:03.000 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:02.998+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.000 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:02.998+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=631795057}) 2024-09-13T11:57:03.000 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:02.998+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1040625421}) 2024-09-13T11:57:03.001 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:02.999+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.001 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:02.999+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1137912670}) 2024-09-13T11:57:03.001 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:02.999+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=277565316}) 2024-09-13T11:57:03.001 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:02.999+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.001 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:02.999+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1137912670}) 2024-09-13T11:57:03.001 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:02.999+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3133832630}) 2024-09-13T11:57:03.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.999+0000 7f9e074b6640 1 -- 172.21.15.100:0/2176512379 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9de403dfa0 msgr2=0x7f9de4040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:03.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.999+0000 7f9e074b6640 1 --2- 172.21.15.100:0/2176512379 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9de403dfa0 0x7f9de4040460 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc005880 tx=0x7f9dfc009040 comp rx=0 tx=0).stop 2024-09-13T11:57:03.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.999+0000 7f9e074b6640 10 monclient: shutdown 2024-09-13T11:57:03.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.999+0000 7f9e074b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:03.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.999+0000 7f9e074b6640 1 -- 172.21.15.100:0/2176512379 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e000a2860 msgr2=0x7f9e00137900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:03.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.999+0000 7f9e074b6640 1 --2- 172.21.15.100:0/2176512379 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e000a2860 0x7f9e00137900 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9df0000c00 tx=0x7f9df00046c0 comp rx=0 tx=0).stop 2024-09-13T11:57:03.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.999+0000 7f9e074b6640 1 -- 172.21.15.100:0/2176512379 shutdown_connections 2024-09-13T11:57:03.003 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.999+0000 7f9e074b6640 1 --2- 172.21.15.100:0/2176512379 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9de403dfa0 0x7f9de4040460 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:03.003 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.999+0000 7f9e074b6640 1 --2- 172.21.15.100:0/2176512379 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e000a2860 0x7f9e00137900 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:03.003 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.999+0000 7f9e074b6640 1 --2- 172.21.15.100:0/2176512379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e0013c9f0 0x7f9e0009eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:03.003 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.999+0000 7f9e074b6640 1 --2- 172.21.15.100:0/2176512379 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e0006ea50 0x7f9e001cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:03.003 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.999+0000 7f9e074b6640 1 -- 172.21.15.100:0/2176512379 >> 172.21.15.100:0/2176512379 conn(0x7f9e000a68e0 msgr2=0x7f9e0013dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:03.003 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.999+0000 7f9e074b6640 1 -- 172.21.15.100:0/2176512379 shutdown_connections 2024-09-13T11:57:03.003 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:02.999+0000 7f9e074b6640 1 -- 172.21.15.100:0/2176512379 wait complete. 2024-09-13T11:57:03.007 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:02.999+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.007 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:02.999+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2090436493}) 2024-09-13T11:57:03.007 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:02.999+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=357353037}) 2024-09-13T11:57:03.008 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:02.999+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.008 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:02.999+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2090436493}) 2024-09-13T11:57:03.008 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:02.999+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3368084745}) 2024-09-13T11:57:03.008 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.000+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.008 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.000+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3777099930}) 2024-09-13T11:57:03.008 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.000+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3326742658}) 2024-09-13T11:57:03.008 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.000+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.008 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.000+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3777099930}) 2024-09-13T11:57:03.008 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.000+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=331122262}) 2024-09-13T11:57:03.008 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.001+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.008 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.001+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1448342505}) 2024-09-13T11:57:03.008 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.001+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1011823520}) 2024-09-13T11:57:03.008 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.001+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.009 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.001+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1448342505}) 2024-09-13T11:57:03.009 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.001+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1193724254}) 2024-09-13T11:57:03.009 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.002+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.009 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.002+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2608410266}) 2024-09-13T11:57:03.009 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.002+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1648520430}) 2024-09-13T11:57:03.009 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.002+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.009 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.002+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2608410266}) 2024-09-13T11:57:03.009 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.002+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=489318020}) 2024-09-13T11:57:03.009 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.002+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.009 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.002+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3513717015}) 2024-09-13T11:57:03.009 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.002+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1300360730}) 2024-09-13T11:57:03.009 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.002+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.009 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.002+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3513717015}) 2024-09-13T11:57:03.010 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.002+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3175038649}) 2024-09-13T11:57:03.010 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.003+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.010 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.003+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3866359519}) 2024-09-13T11:57:03.010 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.003+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=385139907}) 2024-09-13T11:57:03.010 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.003+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.010 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.003+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3866359519}) 2024-09-13T11:57:03.010 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.003+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3007749266}) 2024-09-13T11:57:03.010 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.004+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.010 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.004+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=991894893}) 2024-09-13T11:57:03.010 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.004+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1614920028}) 2024-09-13T11:57:03.010 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.004+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.010 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.004+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=991894893}) 2024-09-13T11:57:03.010 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.004+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2054068127}) 2024-09-13T11:57:03.011 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.005+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.011 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.005+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=27,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=10,mon_config_key=19,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=19} crc {logm=1104678049,mdsmap=2266422338,mgr=4016471625,mgr_metadata=4154780615,mgrstat=2849388642,mon_config_key=617823848,monmap=157995306,nvmeofgw=598337687,osd_metadata=106836248,osd_pg_creating=2112166155,osd_snap=841190635,osdmap=574453015}) 2024-09-13T11:57:03.011 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.005+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=29,mdsmap=2,mgr=8,mgr_command_descs=1,mgrstat=14,mon_config_key=19,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=13} crc {logm=3345640462,mdsmap=3140514009,mgr=2459773887,mgr_command_descs=3192186153,mgrstat=3613976002,mon_config_key=4064232885,monmap=157995306,nvmeofgw=1596038039,osd_metadata=789425444,osd_pg_creating=2112166155,osd_snap=841190635,osdmap=2909613734}) 2024-09-13T11:57:03.011 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.005+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.011 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.005+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=27,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=10,mon_config_key=19,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=19} crc {logm=1104678049,mdsmap=2266422338,mgr=4016471625,mgr_metadata=4154780615,mgrstat=2849388642,mon_config_key=617823848,monmap=157995306,nvmeofgw=598337687,osd_metadata=106836248,osd_pg_creating=2112166155,osd_snap=841190635,osdmap=574453015}) 2024-09-13T11:57:03.011 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.005+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=29,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=18,monmap=2,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=15} crc {logm=1387717834,mdsmap=3458900325,mgr=2459773887,mgr_command_descs=3192186153,mgr_metadata=4154780615,mgrstat=4111848550,mon_config_key=2492129427,monmap=1780612232,nvmeofgw=3517310312,osd_metadata=3351853923,osd_pg_creating=2112166155,osd_snap=841190635,osdmap=2343644202}) 2024-09-13T11:57:03.011 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.006+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.011 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.006+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2779292588}) 2024-09-13T11:57:03.011 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.006+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1626192328}) 2024-09-13T11:57:03.011 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.006+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.011 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.006+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2779292588}) 2024-09-13T11:57:03.011 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.006+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3614778190}) 2024-09-13T11:57:03.013 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.007+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.013 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.007+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2233743011}) 2024-09-13T11:57:03.013 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.007+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=22263279}) 2024-09-13T11:57:03.013 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.007+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.013 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.007+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2233743011}) 2024-09-13T11:57:03.013 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.007+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1474792373}) 2024-09-13T11:57:03.013 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.008+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.013 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.008+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=427595091}) 2024-09-13T11:57:03.013 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.008+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3061825231}) 2024-09-13T11:57:03.013 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.008+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.014 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.008+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=427595091}) 2024-09-13T11:57:03.014 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.008+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4156206281}) 2024-09-13T11:57:03.014 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-13T11:57:03.015 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.012+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.015 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.012+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=17} crc {osdmap=3920166914}) 2024-09-13T11:57:03.015 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.012+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=17} crc {osdmap=3244758696}) 2024-09-13T11:57:03.015 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.012+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:57:03.015 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.012+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=17} crc {osdmap=3920166914}) 2024-09-13T11:57:03.015 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:57:03.012+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=18} crc {osdmap=2613486838}) 2024-09-13T11:57:03.187 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:03.187 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38049]: starting. 2024-09-13T11:57:03.187 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38049]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-38014 2024-09-13T11:57:03.187 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38049]: created object 0... 2024-09-13T11:57:03.187 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38049]: created object 25... 2024-09-13T11:57:03.187 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38049]: created object 49... 2024-09-13T11:57:03.187 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38049]: finishing. 2024-09-13T11:57:03.187 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38049]: shutting down. 2024-09-13T11:57:03.198 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:03.198 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38051]: starting. 2024-09-13T11:57:03.198 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38051]: listing objects. 2024-09-13T11:57:03.198 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38051]: listed object 0... 2024-09-13T11:57:03.198 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38051]: listed object 25... 2024-09-13T11:57:03.198 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38051]: saw 50 objects 2024-09-13T11:57:03.198 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38051]: shutting down. 2024-09-13T11:57:04.015 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-13T11:57:04.015 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:57:04.035 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:04.035 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38050]: starting. 2024-09-13T11:57:04.035 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38050]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-38014 2024-09-13T11:57:04.035 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38050]: shutting down. 2024-09-13T11:57:04.037 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:04.037 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:04.037 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:57:04.041 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:57:04.124 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.122+0000 7fc47b2ef640 10 monclient: get_monmap_and_config 2024-09-13T11:57:04.124 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.122+0000 7fc47b2ef640 10 monclient: build_initial_monmap 2024-09-13T11:57:04.124 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.122+0000 7fc47b2ef640 10 monclient: monmap: 2024-09-13T11:57:04.124 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:04.124 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.125 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:57:04.123074+0000 2024-09-13T11:57:04.125 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:57:04.123074+0000 2024-09-13T11:57:04.125 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:04.125 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.125 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:57:04.126 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:57:04.126 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:57:04.126 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:57:04.126 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:57:04.126 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.122+0000 7fc47b2ef640 1 Processor -- start 2024-09-13T11:57:04.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.122+0000 7fc47b2ef640 1 -- start start 2024-09-13T11:57:04.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.122+0000 7fc47b2ef640 10 monclient: init 2024-09-13T11:57:04.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47b2ef640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:04.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47b2ef640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-13T11:57:04.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47b2ef640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4740a15a0 0x7fc4740a1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47b2ef640 10 monclient(hunting): picked mon.noname-b con 0x7fc4740a15a0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:04.147 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47b2ef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc47409c4e0 addr v1:172.21.15.100:6789/0 2024-09-13T11:57:04.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47b2ef640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc47409be20 0x7fc4740a1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47b2ef640 10 monclient(hunting): picked mon.noname-c con 0x7fc47409be20 addr v2:172.21.15.122:3300/0 2024-09-13T11:57:04.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47b2ef640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc47409f550 con 0x7fc47409c4e0 2024-09-13T11:57:04.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47b2ef640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47b2ef640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7fc47409ed20 con 0x7fc47409be20 2024-09-13T11:57:04.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47b2ef640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47b2ef640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fc47409c2a0 con 0x7fc4740a15a0 2024-09-13T11:57:04.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47b2ef640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:04.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47b2ef640 10 monclient(hunting): authenticate will time out at 1004.716537s 2024-09-13T11:57:04.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47908b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4740a15a0 0x7fc4740a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47908b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4740a15a0 0x7fc4740a1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:46108/0 (socket says 172.21.15.100:46108) 2024-09-13T11:57:04.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47908b640 1 -- 172.21.15.100:0/854450245 learned_addr learned my addr 172.21.15.100:0/854450245 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:04.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47908b640 10 monclient(hunting): get_auth_request con 0x7fc4740a15a0 auth_method 0 2024-09-13T11:57:04.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47888a640 1 --1- 172.21.15.100:0/854450245 >> v1:172.21.15.100:6789/0 conn(0x7fc47409c4e0 0x7fc474135370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:47468/0 (socket says 172.21.15.100:47468) 2024-09-13T11:57:04.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47908b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47908b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47908b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47908b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47908b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47908b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47988c640 1 --2- 172.21.15.100:0/854450245 >> v2:172.21.15.122:3300/0 conn(0x7fc47409be20 0x7fc4740a1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47988c640 10 monclient(hunting): get_auth_request con 0x7fc47409be20 auth_method 0 2024-09-13T11:57:04.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47988c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47988c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47988c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47988c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47988c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47988c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47908b640 10 monclient(hunting): handle_auth_done global_id 24128 payload 274 2024-09-13T11:57:04.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47908b640 1 -- 172.21.15.100:0/854450245 >> v2:172.21.15.122:3300/0 conn(0x7fc47409be20 msgr2=0x7fc4740a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:57:04.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47908b640 1 --2- 172.21.15.100:0/854450245 >> v2:172.21.15.122:3300/0 conn(0x7fc47409be20 0x7fc4740a1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47908b640 1 -- 172.21.15.100:0/854450245 >> v1:172.21.15.100:6789/0 conn(0x7fc47409c4e0 legacy=0x7fc474135370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47908b640 10 monclient: _finish_hunting 0 2024-09-13T11:57:04.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47908b640 1 monclient: found mon.noname-b 2024-09-13T11:57:04.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47908b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:04.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47908b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:57:04.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc46bfff640 1 -- 172.21.15.100:0/854450245 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 173708633 0 0) 0x7fc47409f550 con 0x7fc47409c4e0 2024-09-13T11:57:04.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc47908b640 1 -- 172.21.15.100:0/854450245 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc47409fdf0 con 0x7fc4740a15a0 2024-09-13T11:57:04.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.123+0000 7fc46bfff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:57:04.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.124+0000 7fc47908b640 1 --2- 172.21.15.100:0/854450245 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4740a15a0 0x7fc4740a1980 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc470009870 tx=0x7fc47002a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5746731b4745b892 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:04.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.124+0000 7fc46bfff640 1 -- 172.21.15.100:0/854450245 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc470037070 con 0x7fc4740a15a0 2024-09-13T11:57:04.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.124+0000 7fc46bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.124+0000 7fc46bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:57:04.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.124+0000 7fc46bfff640 10 monclient: dump: 2024-09-13T11:57:04.156 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.156 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.156 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.157 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.157 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.157 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.157 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.157 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.157 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.158 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.124+0000 7fc46bfff640 10 monclient: _finish_auth 0 2024-09-13T11:57:04.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.124+0000 7fc46bfff640 10 monclient: _check_auth_tickets 2024-09-13T11:57:04.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.124+0000 7fc46bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:04.159 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.124+0000 7fc47b2ef640 5 monclient: authenticate success, global_id 24128 2024-09-13T11:57:04.159 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.124+0000 7fc46bfff640 1 -- 172.21.15.100:0/854450245 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc470032440 con 0x7fc4740a15a0 2024-09-13T11:57:04.159 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.124+0000 7fc46bfff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:04.159 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.124+0000 7fc46bfff640 1 -- 172.21.15.100:0/854450245 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc470032810 con 0x7fc4740a15a0 2024-09-13T11:57:04.160 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.124+0000 7fc47b2ef640 10 monclient: get_monmap_and_config success 2024-09-13T11:57:04.160 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.124+0000 7fc47b2ef640 4 set_mon_vals no callback set 2024-09-13T11:57:04.160 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.124+0000 7fc46bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.160 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.124+0000 7fc46bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:57:04.160 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.124+0000 7fc46bfff640 10 monclient: dump: 2024-09-13T11:57:04.160 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.161 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.161 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.161 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.161 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.161 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.162 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.162 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.162 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.162 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.162 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.125+0000 7fc47b2ef640 10 monclient: shutdown 2024-09-13T11:57:04.162 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.125+0000 7fc47b2ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:04.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.125+0000 7fc47b2ef640 1 -- 172.21.15.100:0/854450245 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4740a15a0 msgr2=0x7fc4740a1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.125+0000 7fc47b2ef640 1 --2- 172.21.15.100:0/854450245 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4740a15a0 0x7fc4740a1980 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc470009870 tx=0x7fc47002a130 comp rx=0 tx=0).stop 2024-09-13T11:57:04.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.125+0000 7fc47b2ef640 1 -- 172.21.15.100:0/854450245 shutdown_connections 2024-09-13T11:57:04.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.125+0000 7fc47b2ef640 1 --2- 172.21.15.100:0/854450245 >> v2:172.21.15.122:3300/0 conn(0x7fc47409be20 0x7fc4740a1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.125+0000 7fc47b2ef640 1 --2- 172.21.15.100:0/854450245 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc4740a15a0 0x7fc4740a1980 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.164 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.125+0000 7fc47b2ef640 1 -- 172.21.15.100:0/854450245 >> 172.21.15.100:0/854450245 conn(0x7fc4740a6470 msgr2=0x7fc4740a2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:04.164 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.125+0000 7fc47b2ef640 1 -- 172.21.15.100:0/854450245 shutdown_connections 2024-09-13T11:57:04.164 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.125+0000 7fc47b2ef640 1 -- 172.21.15.100:0/854450245 wait complete. 2024-09-13T11:57:04.164 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.126+0000 7fc47b2ef640 10 monclient: build_initial_monmap 2024-09-13T11:57:04.164 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.126+0000 7fc47b2ef640 10 monclient: monmap: 2024-09-13T11:57:04.165 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:04.165 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:57:04.165 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:57:04.165 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:57:04.165 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:04.165 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.166 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:57:04.166 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:57:04.166 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:57:04.166 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.166 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.126+0000 7fc47b2ef640 1 Processor -- start 2024-09-13T11:57:04.167 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.126+0000 7fc47b2ef640 1 -- start start 2024-09-13T11:57:04.167 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.126+0000 7fc47b2ef640 10 monclient: init 2024-09-13T11:57:04.167 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47b2ef640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:04.167 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47b2ef640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:57:04.167 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47b2ef640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc47409be20 0x7fc47409cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.167 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47b2ef640 10 monclient(hunting): picked mon.noname-b con 0x7fc47409be20 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:57:04.167 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47b2ef640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc47409c4e0 0x7fc47409d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.167 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47888a640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc47409c4e0 0x7fc47409d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.167 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47888a640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc47409c4e0 0x7fc47409d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:46122/0 (socket says 172.21.15.100:46122) 2024-09-13T11:57:04.167 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47888a640 1 -- 172.21.15.100:0/727503570 learned_addr learned my addr 172.21.15.100:0/727503570 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:04.167 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47908b640 1 --2- 172.21.15.100:0/727503570 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc47409be20 0x7fc47409cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.167 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47b2ef640 10 monclient(hunting): picked mon.noname-c con 0x7fc47409c4e0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:04.167 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47b2ef640 1 --2- 172.21.15.100:0/727503570 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4740a15a0 0x7fc47409d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47b2ef640 10 monclient(hunting): picked mon.noname-a con 0x7fc4740a15a0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:57:04.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47b2ef640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47b2ef640 1 -- 172.21.15.100:0/727503570 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fc474135a80 con 0x7fc4740a15a0 2024-09-13T11:57:04.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47b2ef640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47b2ef640 1 -- 172.21.15.100:0/727503570 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc4740a2400 con 0x7fc47409be20 2024-09-13T11:57:04.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47b2ef640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47b2ef640 1 -- 172.21.15.100:0/727503570 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fc474135c00 con 0x7fc47409c4e0 2024-09-13T11:57:04.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47b2ef640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:04.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47b2ef640 10 monclient(hunting): authenticate will time out at 1004.720704s 2024-09-13T11:57:04.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47888a640 10 monclient(hunting): get_auth_request con 0x7fc47409c4e0 auth_method 0 2024-09-13T11:57:04.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47888a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47888a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.168 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47888a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47908b640 10 monclient(hunting): get_auth_request con 0x7fc47409be20 auth_method 0 2024-09-13T11:57:04.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47908b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47908b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47908b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47908b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47908b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47908b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47888a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47888a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47888a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.127+0000 7fc47908b640 10 monclient(hunting): handle_auth_done global_id 5674 payload 824 2024-09-13T11:57:04.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc47908b640 1 -- 172.21.15.100:0/727503570 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc47409c4e0 msgr2=0x7fc47409d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc47908b640 1 --2- 172.21.15.100:0/727503570 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc47409c4e0 0x7fc47409d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc47908b640 1 -- 172.21.15.100:0/727503570 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4740a15a0 msgr2=0x7fc47409d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.170 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc47908b640 1 --2- 172.21.15.100:0/727503570 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4740a15a0 0x7fc47409d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.170 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc47908b640 10 monclient: _finish_hunting 0 2024-09-13T11:57:04.170 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc47908b640 1 monclient: found mon.noname-b 2024-09-13T11:57:04.170 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc47908b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:04.170 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc47908b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.122:3300/0 2024-09-13T11:57:04.170 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc47908b640 1 -- 172.21.15.100:0/727503570 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc474127920 con 0x7fc47409be20 2024-09-13T11:57:04.170 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc47908b640 1 --2- 172.21.15.100:0/727503570 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc47409be20 0x7fc47409cb40 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc470031000 tx=0x7fc4700046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:04.170 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc469ffb640 1 -- 172.21.15.100:0/727503570 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc470041020 con 0x7fc47409be20 2024-09-13T11:57:04.170 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc469ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.170 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc469ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:57:04.170 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc469ffb640 10 monclient: dump: 2024-09-13T11:57:04.170 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.170 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.171 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.171 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.171 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.171 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.171 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.171 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.171 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.171 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.171 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc469ffb640 10 monclient: _finish_auth 0 2024-09-13T11:57:04.171 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc469ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:57:04.171 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc469ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:04.171 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc469ffb640 1 -- 172.21.15.100:0/727503570 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc470030640 con 0x7fc47409be20 2024-09-13T11:57:04.171 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc469ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:04.171 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc469ffb640 1 -- 172.21.15.100:0/727503570 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc470037040 con 0x7fc47409be20 2024-09-13T11:57:04.172 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc47b2ef640 5 monclient: authenticate success, global_id 5674 2024-09-13T11:57:04.172 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc47b2ef640 10 monclient: _renew_subs 2024-09-13T11:57:04.172 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc47b2ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:57:04.172 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc47b2ef640 1 -- 172.21.15.100:0/727503570 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc474072b60 con 0x7fc47409be20 2024-09-13T11:57:04.172 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc469ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.172 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc469ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:57:04.172 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc46b7fe640 4 set_mon_vals no callback set 2024-09-13T11:57:04.172 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.128+0000 7fc469ffb640 10 monclient: dump: 2024-09-13T11:57:04.172 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.172 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.172 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.172 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.172 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.173 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.173 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.173 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.173 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.173 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.173 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.129+0000 7fc47b2ef640 10 monclient: _renew_subs 2024-09-13T11:57:04.173 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.129+0000 7fc47b2ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:57:04.173 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.129+0000 7fc47b2ef640 1 -- 172.21.15.100:0/727503570 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc474127360 con 0x7fc47409be20 2024-09-13T11:57:04.173 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.129+0000 7fc469ffb640 1 -- 172.21.15.100:0/727503570 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fc470044020 con 0x7fc47409be20 2024-09-13T11:57:04.173 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.130+0000 7fc47b2ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:04.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.130+0000 7fc47b2ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:04.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.130+0000 7fc47b2ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:57:04.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.130+0000 7fc469ffb640 1 --2- 172.21.15.100:0/727503570 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc45803dfa0 0x7fc458040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.130+0000 7fc47b2ef640 1 -- 172.21.15.100:0/727503570 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc474136030 con 0x7fc47409be20 2024-09-13T11:57:04.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.130+0000 7fc469ffb640 1 -- 172.21.15.100:0/727503570 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(223..223 src has 1..223) ==== 4346+0+0 (secure 0 0 0) 0x7fc470072c50 con 0x7fc47409be20 2024-09-13T11:57:04.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.130+0000 7fc47888a640 1 --2- 172.21.15.100:0/727503570 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc45803dfa0 0x7fc458040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.130+0000 7fc47888a640 10 monclient: get_auth_request con 0x7fc45803dfa0 auth_method 0 2024-09-13T11:57:04.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.130+0000 7fc47888a640 1 --2- 172.21.15.100:0/727503570 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc45803dfa0 0x7fc458040460 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc474126000 tx=0x7fc464007d70 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:04.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.135+0000 7fc469ffb640 1 -- 172.21.15.100:0/727503570 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc47004b030 con 0x7fc47409be20 2024-09-13T11:57:04.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.135+0000 7fc469ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:04.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.135+0000 7fc469ffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:57:04.379 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.377+0000 7fc47b2ef640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:57:04.379 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.377+0000 7fc47b2ef640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:57:04.379 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.377+0000 7fc47b2ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.122:3300/0 2024-09-13T11:57:04.379 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.377+0000 7fc47b2ef640 1 -- 172.21.15.100:0/727503570 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc474127d40 con 0x7fc47409be20 2024-09-13T11:57:04.379 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.378+0000 7fc469ffb640 1 -- 172.21.15.100:0/727503570 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fc47003c380 con 0x7fc47409be20 2024-09-13T11:57:04.379 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.378+0000 7fc469ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:57:04.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.378+0000 7fc469ffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:57:04.380 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:57:04.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.380+0000 7fc47b2ef640 1 -- 172.21.15.100:0/727503570 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc45803dfa0 msgr2=0x7fc458040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.380+0000 7fc47b2ef640 1 --2- 172.21.15.100:0/727503570 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc45803dfa0 0x7fc458040460 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc474126000 tx=0x7fc464007d70 comp rx=0 tx=0).stop 2024-09-13T11:57:04.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.380+0000 7fc47b2ef640 10 monclient: shutdown 2024-09-13T11:57:04.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.380+0000 7fc47b2ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:04.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.380+0000 7fc47b2ef640 1 -- 172.21.15.100:0/727503570 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc47409be20 msgr2=0x7fc47409cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.380+0000 7fc47b2ef640 1 --2- 172.21.15.100:0/727503570 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc47409be20 0x7fc47409cb40 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc470031000 tx=0x7fc4700046c0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.380+0000 7fc47b2ef640 1 -- 172.21.15.100:0/727503570 shutdown_connections 2024-09-13T11:57:04.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.380+0000 7fc47b2ef640 1 --2- 172.21.15.100:0/727503570 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc45803dfa0 0x7fc458040460 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.380+0000 7fc47b2ef640 1 --2- 172.21.15.100:0/727503570 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc4740a15a0 0x7fc47409d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.380+0000 7fc47b2ef640 1 --2- 172.21.15.100:0/727503570 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc47409c4e0 0x7fc47409d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.380+0000 7fc47b2ef640 1 --2- 172.21.15.100:0/727503570 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc47409be20 0x7fc47409cb40 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.380+0000 7fc47b2ef640 1 -- 172.21.15.100:0/727503570 >> 172.21.15.100:0/727503570 conn(0x7fc4740a6470 msgr2=0x7fc4740a5180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:04.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.380+0000 7fc47b2ef640 1 -- 172.21.15.100:0/727503570 shutdown_connections 2024-09-13T11:57:04.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.380+0000 7fc47b2ef640 1 -- 172.21.15.100:0/727503570 wait complete. 2024-09-13T11:57:04.392 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-13T11:57:04.392 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-13T11:57:04.392 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-13T11:57:04.392 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-13T11:57:04.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.488+0000 7f7e1b660640 10 monclient: get_monmap_and_config 2024-09-13T11:57:04.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.488+0000 7f7e1b660640 10 monclient: build_initial_monmap 2024-09-13T11:57:04.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.488+0000 7f7e1b660640 10 monclient: monmap: 2024-09-13T11:57:04.490 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:04.490 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.490 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:57:04.489427+0000 2024-09-13T11:57:04.490 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:57:04.489427+0000 2024-09-13T11:57:04.490 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:04.491 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.491 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:57:04.491 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:57:04.491 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:57:04.491 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:57:04.491 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:57:04.491 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.489+0000 7f7e1b660640 1 Processor -- start 2024-09-13T11:57:04.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.489+0000 7f7e1b660640 1 -- start start 2024-09-13T11:57:04.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.489+0000 7f7e1b660640 10 monclient: init 2024-09-13T11:57:04.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.489+0000 7f7e1b660640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:04.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.489+0000 7f7e1b660640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-13T11:57:04.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.489+0000 7f7e1b660640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e14073550 0x7f7e14073930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.489+0000 7f7e1b660640 10 monclient(hunting): picked mon.noname-b con 0x7f7e14073550 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:04.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.489+0000 7f7e1b660640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7e140a1fe0 0x7f7e140a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.489+0000 7f7e1b660640 10 monclient(hunting): picked mon.noname-c con 0x7f7e140a1fe0 addr v2:172.21.15.122:3300/0 2024-09-13T11:57:04.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.489+0000 7f7e1b660640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7e140a3d10 0x7f7e140a41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.489+0000 7f7e1b660640 10 monclient(hunting): picked mon.noname-a con 0x7f7e140a3d10 addr v2:172.21.15.100:3300/0 2024-09-13T11:57:04.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.489+0000 7f7e1b660640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.489+0000 7f7e1b660640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f7e1409fda0 con 0x7f7e140a3d10 2024-09-13T11:57:04.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.489+0000 7f7e1b660640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.489+0000 7f7e1b660640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f7e140a0db0 con 0x7f7e140a1fe0 2024-09-13T11:57:04.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.489+0000 7f7e1b660640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.489+0000 7f7e1b660640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f7e140a1020 con 0x7f7e14073550 2024-09-13T11:57:04.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.489+0000 7f7e1b660640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:04.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.489+0000 7f7e1b660640 10 monclient(hunting): authenticate will time out at 1005.082876s 2024-09-13T11:57:04.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.489+0000 7f7e193fc640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e14073550 0x7f7e14073930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e193fc640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e14073550 0x7f7e14073930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:46124/0 (socket says 172.21.15.100:46124) 2024-09-13T11:57:04.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e19bfd640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7e140a3d10 0x7f7e140a41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e193fc640 1 -- 172.21.15.100:0/2991050436 learned_addr learned my addr 172.21.15.100:0/2991050436 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:04.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e193fc640 10 monclient(hunting): get_auth_request con 0x7f7e14073550 auth_method 0 2024-09-13T11:57:04.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e193fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e193fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e193fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e19bfd640 10 monclient(hunting): get_auth_request con 0x7f7e140a3d10 auth_method 0 2024-09-13T11:57:04.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e19bfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e19bfd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e19bfd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e193fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e193fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e193fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e19bfd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e19bfd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e19bfd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e18bfb640 1 --2- 172.21.15.100:0/2991050436 >> v2:172.21.15.122:3300/0 conn(0x7f7e140a1fe0 0x7f7e140a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e193fc640 10 monclient(hunting): handle_auth_done global_id 24134 payload 274 2024-09-13T11:57:04.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e193fc640 1 -- 172.21.15.100:0/2991050436 >> v2:172.21.15.122:3300/0 conn(0x7f7e140a1fe0 msgr2=0x7f7e140a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:57:04.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e193fc640 1 --2- 172.21.15.100:0/2991050436 >> v2:172.21.15.122:3300/0 conn(0x7f7e140a1fe0 0x7f7e140a37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e193fc640 1 -- 172.21.15.100:0/2991050436 >> v2:172.21.15.100:3300/0 conn(0x7f7e140a3d10 msgr2=0x7f7e140a41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:57:04.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e193fc640 1 --2- 172.21.15.100:0/2991050436 >> v2:172.21.15.100:3300/0 conn(0x7f7e140a3d10 0x7f7e140a41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e193fc640 10 monclient: _finish_hunting 0 2024-09-13T11:57:04.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e193fc640 1 monclient: found mon.noname-b 2024-09-13T11:57:04.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e193fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:04.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e193fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:57:04.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e193fc640 1 -- 172.21.15.100:0/2991050436 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e140a1300 con 0x7f7e14073550 2024-09-13T11:57:04.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e19bfd640 1 --2- 172.21.15.100:0/2991050436 >> v2:172.21.15.100:3300/0 conn(0x7f7e140a3d10 0x7f7e140a41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:57:04.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e18bfb640 10 monclient: get_auth_request con 0x7f7e140a1fe0 auth_method 0 2024-09-13T11:57:04.506 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e18bfb640 1 --2- 172.21.15.100:0/2991050436 >> v2:172.21.15.122:3300/0 conn(0x7f7e140a1fe0 0x7f7e140a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:57:04.506 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.490+0000 7f7e193fc640 1 --2- 172.21.15.100:0/2991050436 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e14073550 0x7f7e14073930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7e140a5770 tx=0x7f7e1002a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2b9565180c16c93 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:04.506 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e0bfff640 1 -- 172.21.15.100:0/2991050436 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e10037070 con 0x7f7e14073550 2024-09-13T11:57:04.506 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.506 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e0bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:57:04.506 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e0bfff640 10 monclient: dump: 2024-09-13T11:57:04.506 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.506 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.506 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.506 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.506 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.506 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.506 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.507 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.507 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.507 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.507 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e0bfff640 10 monclient: _finish_auth 0 2024-09-13T11:57:04.507 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e0bfff640 10 monclient: _check_auth_tickets 2024-09-13T11:57:04.507 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:04.507 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e1b660640 5 monclient: authenticate success, global_id 24134 2024-09-13T11:57:04.507 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e1b660640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:57:04.507 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e0bfff640 1 -- 172.21.15.100:0/2991050436 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e1002ae10 con 0x7f7e14073550 2024-09-13T11:57:04.507 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e0bfff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:04.507 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e0bfff640 1 -- 172.21.15.100:0/2991050436 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e10032660 con 0x7f7e14073550 2024-09-13T11:57:04.507 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e1b660640 10 monclient: get_monmap_and_config success 2024-09-13T11:57:04.508 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e1b660640 4 set_mon_vals no callback set 2024-09-13T11:57:04.508 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.508 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e0bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:57:04.508 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e0bfff640 10 monclient: dump: 2024-09-13T11:57:04.508 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.508 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.508 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.508 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.508 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.508 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.508 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.509 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.509 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.509 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.509 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e1b660640 10 monclient: shutdown 2024-09-13T11:57:04.509 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e1b660640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:04.509 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e1b660640 1 -- 172.21.15.100:0/2991050436 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e14073550 msgr2=0x7f7e14073930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.509 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e1b660640 1 --2- 172.21.15.100:0/2991050436 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e14073550 0x7f7e14073930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7e140a5770 tx=0x7f7e1002a100 comp rx=0 tx=0).stop 2024-09-13T11:57:04.509 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e1b660640 1 -- 172.21.15.100:0/2991050436 shutdown_connections 2024-09-13T11:57:04.509 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e1b660640 1 --2- 172.21.15.100:0/2991050436 >> v2:172.21.15.100:3300/0 conn(0x7f7e140a3d10 0x7f7e140a41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.509 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e1b660640 1 --2- 172.21.15.100:0/2991050436 >> v2:172.21.15.122:3300/0 conn(0x7f7e140a1fe0 0x7f7e140a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.509 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e1b660640 1 --2- 172.21.15.100:0/2991050436 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e14073550 0x7f7e14073930 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.509 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e1b660640 1 -- 172.21.15.100:0/2991050436 >> 172.21.15.100:0/2991050436 conn(0x7f7e140a83f0 msgr2=0x7f7e140a48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:04.510 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e1b660640 1 -- 172.21.15.100:0/2991050436 shutdown_connections 2024-09-13T11:57:04.510 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.491+0000 7f7e1b660640 1 -- 172.21.15.100:0/2991050436 wait complete. 2024-09-13T11:57:04.510 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.492+0000 7f7e1b660640 10 monclient: build_initial_monmap 2024-09-13T11:57:04.510 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.492+0000 7f7e1b660640 10 monclient: monmap: 2024-09-13T11:57:04.510 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:04.510 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:57:04.510 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:57:04.510 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:57:04.510 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:04.510 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.510 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:57:04.510 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:57:04.510 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:57:04.511 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.511 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.492+0000 7f7e1b660640 1 Processor -- start 2024-09-13T11:57:04.511 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.492+0000 7f7e1b660640 1 -- start start 2024-09-13T11:57:04.511 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.492+0000 7f7e1b660640 10 monclient: init 2024-09-13T11:57:04.511 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e1b660640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:04.511 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e1b660640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:57:04.511 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e1b660640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e14073550 0x7f7e141ac750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.511 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e1b660640 10 monclient(hunting): picked mon.noname-a con 0x7f7e14073550 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:57:04.511 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e1b660640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7e140a1fe0 0x7f7e141ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.511 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e1b660640 10 monclient(hunting): picked mon.noname-b con 0x7f7e140a1fe0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:57:04.511 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e1b660640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e140a3d10 0x7f7e141c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.511 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e1b660640 10 monclient(hunting): picked mon.noname-c con 0x7f7e140a3d10 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:04.511 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e1b660640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e1b660640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f7e14073ff0 con 0x7f7e14073550 2024-09-13T11:57:04.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e1b660640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e1b660640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f7e14073e70 con 0x7f7e140a1fe0 2024-09-13T11:57:04.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e1b660640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e1b660640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f7e140a5240 con 0x7f7e140a3d10 2024-09-13T11:57:04.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e1b660640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:04.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e1b660640 10 monclient(hunting): authenticate will time out at 1005.086180s 2024-09-13T11:57:04.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e193fc640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e14073550 0x7f7e141ac750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e193fc640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e14073550 0x7f7e141ac750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55436/0 (socket says 172.21.15.100:55436) 2024-09-13T11:57:04.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e193fc640 1 -- 172.21.15.100:0/276077588 learned_addr learned my addr 172.21.15.100:0/276077588 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:04.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e19bfd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e140a3d10 0x7f7e141c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e193fc640 10 monclient(hunting): get_auth_request con 0x7f7e14073550 auth_method 0 2024-09-13T11:57:04.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e193fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e193fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e193fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e19bfd640 10 monclient(hunting): get_auth_request con 0x7f7e140a3d10 auth_method 0 2024-09-13T11:57:04.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e19bfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e19bfd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e19bfd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e193fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e193fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e193fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e19bfd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e19bfd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e19bfd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e18bfb640 1 --2- 172.21.15.100:0/276077588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7e140a1fe0 0x7f7e141ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e18bfb640 10 monclient(hunting): get_auth_request con 0x7f7e140a1fe0 auth_method 0 2024-09-13T11:57:04.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e18bfb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e18bfb640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e18bfb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e193fc640 10 monclient(hunting): handle_auth_done global_id 14904 payload 824 2024-09-13T11:57:04.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e193fc640 1 -- 172.21.15.100:0/276077588 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e140a3d10 msgr2=0x7f7e141c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e193fc640 1 --2- 172.21.15.100:0/276077588 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e140a3d10 0x7f7e141c65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e193fc640 1 -- 172.21.15.100:0/276077588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7e140a1fe0 msgr2=0x7f7e141ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e193fc640 1 --2- 172.21.15.100:0/276077588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7e140a1fe0 0x7f7e141ccaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e193fc640 10 monclient: _finish_hunting 0 2024-09-13T11:57:04.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e193fc640 1 monclient: found mon.noname-a 2024-09-13T11:57:04.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e193fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:04.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e193fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:57:04.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e193fc640 1 -- 172.21.15.100:0/276077588 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e141ad0e0 con 0x7f7e14073550 2024-09-13T11:57:04.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e19bfd640 1 --2- 172.21.15.100:0/276077588 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e140a3d10 0x7f7e141c65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:57:04.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e193fc640 1 --2- 172.21.15.100:0/276077588 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e14073550 0x7f7e141ac750 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7e10000c00 tx=0x7f7e100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:04.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e09ffb640 1 -- 172.21.15.100:0/276077588 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e10041070 con 0x7f7e14073550 2024-09-13T11:57:04.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e09ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:57:04.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e09ffb640 10 monclient: dump: 2024-09-13T11:57:04.515 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.515 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.515 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.515 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.516 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.516 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.516 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.516 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.516 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.516 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e09ffb640 10 monclient: _finish_auth 0 2024-09-13T11:57:04.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e09ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:57:04.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:04.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e1b660640 5 monclient: authenticate success, global_id 14904 2024-09-13T11:57:04.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e09ffb640 1 -- 172.21.15.100:0/276077588 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e1003c460 con 0x7f7e14073550 2024-09-13T11:57:04.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e1b660640 10 monclient: _renew_subs 2024-09-13T11:57:04.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e1b660640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:04.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e1b660640 1 -- 172.21.15.100:0/276077588 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e141ab5e0 con 0x7f7e14073550 2024-09-13T11:57:04.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e09ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:04.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e09ffb640 1 -- 172.21.15.100:0/276077588 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e10037040 con 0x7f7e14073550 2024-09-13T11:57:04.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e09ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:04.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e09ffb640 10 monclient: dump: 2024-09-13T11:57:04.517 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.517 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.517 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.517 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.517 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.517 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.518 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.518 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.518 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.518 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.493+0000 7f7e0b7fe640 4 set_mon_vals no callback set 2024-09-13T11:57:04.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.494+0000 7f7e1b660640 10 monclient: _renew_subs 2024-09-13T11:57:04.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.494+0000 7f7e1b660640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:04.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.494+0000 7f7e1b660640 1 -- 172.21.15.100:0/276077588 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e141abbf0 con 0x7f7e14073550 2024-09-13T11:57:04.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.494+0000 7f7e1b660640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:04.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.494+0000 7f7e1b660640 1 --2- 172.21.15.100:0/276077588 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e141c90d0 0x7f7e141c94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.494+0000 7f7e1b660640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.494+0000 7f7e1b660640 1 -- 172.21.15.100:0/276077588 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f7e141adbb0 con 0x7f7e141c90d0 2024-09-13T11:57:04.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.494+0000 7f7e18bfb640 1 --2- 172.21.15.100:0/276077588 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e141c90d0 0x7f7e141c94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.494+0000 7f7e18bfb640 10 monclient: get_auth_request con 0x7f7e141c90d0 auth_method 0 2024-09-13T11:57:04.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.494+0000 7f7e18bfb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.494+0000 7f7e18bfb640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.494+0000 7f7e18bfb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.495+0000 7f7e18bfb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.495+0000 7f7e18bfb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.495+0000 7f7e18bfb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.495+0000 7f7e09ffb640 1 -- 172.21.15.100:0/276077588 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f7e10044070 con 0x7f7e14073550 2024-09-13T11:57:04.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.495+0000 7f7e18bfb640 10 monclient(hunting): handle_auth_done global_id 14907 payload 824 2024-09-13T11:57:04.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.495+0000 7f7e18bfb640 1 -- 172.21.15.100:0/276077588 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7e14071060 con 0x7f7e141c90d0 2024-09-13T11:57:04.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.495+0000 7f7e09ffb640 1 --2- 172.21.15.100:0/276077588 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f7df803dfd0 0x7f7df8040490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.495+0000 7f7e18bfb640 1 --2- 172.21.15.100:0/276077588 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e141c90d0 0x7f7e141c94b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7e14074790 tx=0x7f7e0400f910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:04.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.495+0000 7f7e09ffb640 1 -- 172.21.15.100:0/276077588 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(223..223 src has 1..223) ==== 4346+0+0 (secure 0 0 0) 0x7f7e10072b10 con 0x7f7e14073550 2024-09-13T11:57:04.520 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.495+0000 7f7e09ffb640 1 -- 172.21.15.100:0/276077588 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e04018070 con 0x7f7e141c90d0 2024-09-13T11:57:04.520 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.495+0000 7f7e09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.520 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.495+0000 7f7e09ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:04.520 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.495+0000 7f7e09ffb640 10 monclient: dump: 2024-09-13T11:57:04.520 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.520 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.520 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.520 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.520 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.520 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.520 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.520 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.520 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.521 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.495+0000 7f7e19bfd640 1 --2- 172.21.15.100:0/276077588 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f7df803dfd0 0x7f7df8040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.495+0000 7f7e19bfd640 10 monclient: get_auth_request con 0x7f7df803dfd0 auth_method 0 2024-09-13T11:57:04.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.495+0000 7f7e09ffb640 1 -- 172.21.15.100:0/276077588 <== mon.0 v2:172.21.15.100:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7e04014890 con 0x7f7e141c90d0 2024-09-13T11:57:04.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.495+0000 7f7e09ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:04.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.495+0000 7f7e09ffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:57:04.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.495+0000 7f7e09ffb640 1 -- 172.21.15.100:0/276077588 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e141c90d0 msgr2=0x7f7e141c94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.495+0000 7f7e09ffb640 1 --2- 172.21.15.100:0/276077588 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e141c90d0 0x7f7e141c94b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7e14074790 tx=0x7f7e0400f910 comp rx=0 tx=0).stop 2024-09-13T11:57:04.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.495+0000 7f7e09ffb640 1 -- 172.21.15.100:0/276077588 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e141c90d0 msgr2=0x7f7e141c94b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:57:04.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.495+0000 7f7e09ffb640 1 --2- 172.21.15.100:0/276077588 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e141c90d0 0x7f7e141c94b0 secure :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7e14074790 tx=0x7f7e0400f910 comp rx=0 tx=0).stop 2024-09-13T11:57:04.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.496+0000 7f7e19bfd640 1 --2- 172.21.15.100:0/276077588 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f7df803dfd0 0x7f7df8040490 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c003c00 tx=0x7f7e0c00b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:04.523 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.501+0000 7f7e1b660640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:57:04.523 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.501+0000 7f7e1b660640 1 --2- 172.21.15.100:0/276077588 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e141c99f0 0x7f7e1406e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.523 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.501+0000 7f7e1b660640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.523 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.501+0000 7f7e1b660640 1 -- 172.21.15.100:0/276077588 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f7e141ae010 con 0x7f7e141c99f0 2024-09-13T11:57:04.523 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.501+0000 7f7e18bfb640 1 --2- 172.21.15.100:0/276077588 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e141c99f0 0x7f7e1406e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.523 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.501+0000 7f7e18bfb640 10 monclient: get_auth_request con 0x7f7e141c99f0 auth_method 0 2024-09-13T11:57:04.523 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.501+0000 7f7e18bfb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.524 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.501+0000 7f7e18bfb640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.524 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.501+0000 7f7e18bfb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.524 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.502+0000 7f7e18bfb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.524 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.502+0000 7f7e18bfb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.524 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.502+0000 7f7e18bfb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.524 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.502+0000 7f7e18bfb640 10 monclient(hunting): handle_auth_done global_id 14910 payload 824 2024-09-13T11:57:04.524 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.502+0000 7f7e18bfb640 1 -- 172.21.15.100:0/276077588 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7e1406ca60 con 0x7f7e141c99f0 2024-09-13T11:57:04.524 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.502+0000 7f7e18bfb640 1 --2- 172.21.15.100:0/276077588 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e141c99f0 0x7f7e1406e950 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7e0400dfd0 tx=0x7f7e040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:04.524 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.502+0000 7f7e09ffb640 1 -- 172.21.15.100:0/276077588 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e04003b70 con 0x7f7e141c99f0 2024-09-13T11:57:04.524 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.502+0000 7f7e09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.524 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.502+0000 7f7e09ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:04.524 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.502+0000 7f7e09ffb640 10 monclient: dump: 2024-09-13T11:57:04.524 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.524 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.525 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.525 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.525 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.525 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.525 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.525 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.525 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.525 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.525 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.502+0000 7f7e09ffb640 1 -- 172.21.15.100:0/276077588 <== mon.0 v2:172.21.15.100:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f7e04003b70 con 0x7f7e141c99f0 2024-09-13T11:57:04.528 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.502+0000 7f7e09ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:57:04.528 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.502+0000 7f7e09ffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:57:04.528 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.502+0000 7f7e09ffb640 1 -- 172.21.15.100:0/276077588 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e141c99f0 msgr2=0x7f7e1406e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.528 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.502+0000 7f7e09ffb640 1 --2- 172.21.15.100:0/276077588 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e141c99f0 0x7f7e1406e950 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7e0400dfd0 tx=0x7f7e040046c0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.528 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.502+0000 7f7e09ffb640 1 -- 172.21.15.100:0/276077588 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e141c99f0 msgr2=0x7f7e1406e950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:57:04.529 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.502+0000 7f7e09ffb640 1 --2- 172.21.15.100:0/276077588 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e141c99f0 0x7f7e1406e950 secure :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7e0400dfd0 tx=0x7f7e040046c0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.529 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:57:04.529 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:57:04.529 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:57:04.529 INFO:teuthology.orchestra.run.smithi100.stdout: "state": "leader", 2024-09-13T11:57:04.529 INFO:teuthology.orchestra.run.smithi100.stdout: "election_epoch": 22, 2024-09-13T11:57:04.529 INFO:teuthology.orchestra.run.smithi100.stdout: "uptime": 125501, 2024-09-13T11:57:04.529 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum": [ 2024-09-13T11:57:04.529 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-13T11:57:04.530 INFO:teuthology.orchestra.run.smithi100.stdout: 1, 2024-09-13T11:57:04.530 INFO:teuthology.orchestra.run.smithi100.stdout: 2 2024-09-13T11:57:04.530 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:04.530 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_age": 6, 2024-09-13T11:57:04.530 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:57:04.530 INFO:teuthology.orchestra.run.smithi100.stdout: "required_con": "2450521708007866372", 2024-09-13T11:57:04.530 INFO:teuthology.orchestra.run.smithi100.stdout: "required_mon": [ 2024-09-13T11:57:04.530 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:57:04.530 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:57:04.530 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:57:04.530 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:57:04.530 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:57:04.530 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:57:04.530 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:57:04.531 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:57:04.531 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:57:04.531 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:57:04.531 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:57:04.531 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:04.531 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:57:04.531 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_mon": [ 2024-09-13T11:57:04.531 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:57:04.531 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:57:04.531 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:57:04.531 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:57:04.531 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:57:04.531 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:57:04.532 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:57:04.532 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:57:04.532 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:57:04.532 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:57:04.532 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:57:04.532 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:04.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.532 INFO:teuthology.orchestra.run.smithi100.stdout: "outside_quorum": [], 2024-09-13T11:57:04.532 INFO:teuthology.orchestra.run.smithi100.stdout: "extra_probe_peers": [], 2024-09-13T11:57:04.532 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_provider": [], 2024-09-13T11:57:04.532 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": { 2024-09-13T11:57:04.532 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-09-13T11:57:04.532 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "0b1f4b4d-fad4-4dd3-86ac-a286d80846c0", 2024-09-13T11:57:04.532 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:57:04.533 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:57:04.533 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release": 19, 2024-09-13T11:57:04.533 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release_name": "squid", 2024-09-13T11:57:04.533 INFO:teuthology.orchestra.run.smithi100.stdout: "election_strategy": 1, 2024-09-13T11:57:04.533 INFO:teuthology.orchestra.run.smithi100.stdout: "disallowed_leaders: ": "", 2024-09-13T11:57:04.533 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false, 2024-09-13T11:57:04.533 INFO:teuthology.orchestra.run.smithi100.stdout: "tiebreaker_mon": "", 2024-09-13T11:57:04.533 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_ranks: ": "", 2024-09-13T11:57:04.533 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:57:04.533 INFO:teuthology.orchestra.run.smithi100.stdout: "persistent": [ 2024-09-13T11:57:04.533 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:57:04.533 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:57:04.533 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:57:04.534 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:57:04.534 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:57:04.534 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:57:04.534 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:57:04.534 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:57:04.534 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:57:04.534 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:57:04.534 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:57:04.534 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:04.534 INFO:teuthology.orchestra.run.smithi100.stdout: "optional": [] 2024-09-13T11:57:04.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.534 INFO:teuthology.orchestra.run.smithi100.stdout: "mons": [ 2024-09-13T11:57:04.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.535 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:57:04.535 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:57:04.535 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:57:04.535 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:57:04.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.535 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:57:04.535 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3300", 2024-09-13T11:57:04.535 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:04.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.535 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:57:04.535 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789", 2024-09-13T11:57:04.536 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:04.536 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:04.536 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:04.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.536 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789/0", 2024-09-13T11:57:04.536 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-13T11:57:04.536 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:57:04.536 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:57:04.536 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:57:04.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.536 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:57:04.537 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:57:04.537 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:57:04.537 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:57:04.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.537 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:57:04.537 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:3300", 2024-09-13T11:57:04.537 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:04.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.537 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:57:04.537 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789", 2024-09-13T11:57:04.537 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:04.537 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:04.537 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:04.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.538 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789/0", 2024-09-13T11:57:04.538 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.122:6789/0", 2024-09-13T11:57:04.538 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:57:04.538 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:57:04.538 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:57:04.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.538 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:57:04.538 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:57:04.538 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:57:04.538 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:57:04.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.539 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:57:04.539 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3301", 2024-09-13T11:57:04.539 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:04.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.539 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:57:04.539 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790", 2024-09-13T11:57:04.539 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:04.539 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:04.539 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:04.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.539 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790/0", 2024-09-13T11:57:04.539 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-13T11:57:04.539 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:57:04.540 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:57:04.540 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:57:04.540 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:04.540 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:04.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.540 INFO:teuthology.orchestra.run.smithi100.stdout: "feature_map": { 2024-09-13T11:57:04.540 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": [ 2024-09-13T11:57:04.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.540 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:04.540 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:04.540 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:57:04.540 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:04.540 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:04.541 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": [ 2024-09-13T11:57:04.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.541 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:04.541 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:04.541 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 2 2024-09-13T11:57:04.541 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:04.541 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:04.541 INFO:teuthology.orchestra.run.smithi100.stdout: "client": [ 2024-09-13T11:57:04.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.541 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:04.541 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:04.541 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 3 2024-09-13T11:57:04.541 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:04.541 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:04.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.542 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false 2024-09-13T11:57:04.542 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:57:04.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.503+0000 7f7e1b660640 1 -- 172.21.15.100:0/276077588 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f7df803dfd0 msgr2=0x7f7df8040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.503+0000 7f7e1b660640 1 --2- 172.21.15.100:0/276077588 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f7df803dfd0 0x7f7df8040490 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c003c00 tx=0x7f7e0c00b040 comp rx=0 tx=0).stop 2024-09-13T11:57:04.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.503+0000 7f7e1b660640 10 monclient: shutdown 2024-09-13T11:57:04.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.503+0000 7f7e1b660640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:04.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.503+0000 7f7e1b660640 1 -- 172.21.15.100:0/276077588 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e14073550 msgr2=0x7f7e141ac750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.503+0000 7f7e1b660640 1 --2- 172.21.15.100:0/276077588 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e14073550 0x7f7e141ac750 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7e10000c00 tx=0x7f7e100046c0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.503+0000 7f7e19bfd640 1 -- 172.21.15.100:0/276077588 reap_dead start 2024-09-13T11:57:04.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.503+0000 7f7e19bfd640 1 -- 172.21.15.100:0/276077588 reap_dead start 2024-09-13T11:57:04.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.503+0000 7f7e1b660640 1 -- 172.21.15.100:0/276077588 shutdown_connections 2024-09-13T11:57:04.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.503+0000 7f7e1b660640 1 -- 172.21.15.100:0/276077588 >> 172.21.15.100:0/276077588 conn(0x7f7e140a83f0 msgr2=0x7f7e14072100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:04.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.503+0000 7f7e1b660640 1 -- 172.21.15.100:0/276077588 shutdown_connections 2024-09-13T11:57:04.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.503+0000 7f7e1b660640 1 -- 172.21.15.100:0/276077588 wait complete. 2024-09-13T11:57:04.544 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T11:57:04.544 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-13T11:57:04.545 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T11:57:04.654 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.641+0000 7fed225f5640 10 monclient: get_monmap_and_config 2024-09-13T11:57:04.654 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.641+0000 7fed225f5640 10 monclient: build_initial_monmap 2024-09-13T11:57:04.654 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.641+0000 7fed225f5640 10 monclient: monmap: 2024-09-13T11:57:04.654 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:04.654 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.654 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:57:04.642008+0000 2024-09-13T11:57:04.654 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:57:04.642008+0000 2024-09-13T11:57:04.655 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:04.655 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.655 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:57:04.655 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:57:04.655 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:57:04.655 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:57:04.655 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:57:04.655 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.655 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.641+0000 7fed225f5640 1 Processor -- start 2024-09-13T11:57:04.655 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.641+0000 7fed225f5640 1 -- start start 2024-09-13T11:57:04.655 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.641+0000 7fed225f5640 10 monclient: init 2024-09-13T11:57:04.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed225f5640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:04.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed225f5640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-13T11:57:04.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed225f5640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c06ea50 0x7fed1c0a2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed225f5640 10 monclient(hunting): picked mon.noname-b con 0x7fed1c06ea50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:04.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed225f5640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fed1c06ee30 0x7fed1c06f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed225f5640 10 monclient(hunting): picked mon.noname-c con 0x7fed1c06ee30 addr v2:172.21.15.122:3300/0 2024-09-13T11:57:04.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed225f5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fed1c06f790 addr v1:172.21.15.122:6789/0 2024-09-13T11:57:04.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed225f5640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed1c0a1590 con 0x7fed1c06f790 2024-09-13T11:57:04.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed225f5640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed225f5640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7fed1c0a1870 con 0x7fed1c06ee30 2024-09-13T11:57:04.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed225f5640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed225f5640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fed1c0a1b50 con 0x7fed1c06ea50 2024-09-13T11:57:04.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed225f5640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:04.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed225f5640 10 monclient(hunting): authenticate will time out at 1005.235453s 2024-09-13T11:57:04.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed1bfff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c06ea50 0x7fed1c0a2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed1bfff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c06ea50 0x7fed1c0a2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:46142/0 (socket says 172.21.15.100:46142) 2024-09-13T11:57:04.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed1bfff640 1 -- 172.21.15.100:0/2284371419 learned_addr learned my addr 172.21.15.100:0/2284371419 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:04.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed1bfff640 10 monclient(hunting): get_auth_request con 0x7fed1c06ea50 auth_method 0 2024-09-13T11:57:04.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.658 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.658 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.658 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed1b7fe640 1 --2- 172.21.15.100:0/2284371419 >> v2:172.21.15.122:3300/0 conn(0x7fed1c06ee30 0x7fed1c06f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.658 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed1b7fe640 10 monclient(hunting): get_auth_request con 0x7fed1c06ee30 auth_method 0 2024-09-13T11:57:04.658 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.658 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.658 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.658 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed1bfff640 10 monclient(hunting): handle_auth_done global_id 24140 payload 274 2024-09-13T11:57:04.658 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed1bfff640 1 -- 172.21.15.100:0/2284371419 >> v2:172.21.15.122:3300/0 conn(0x7fed1c06ee30 msgr2=0x7fed1c06f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:57:04.658 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed1bfff640 1 --2- 172.21.15.100:0/2284371419 >> v2:172.21.15.122:3300/0 conn(0x7fed1c06ee30 0x7fed1c06f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.658 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed1bfff640 1 -- 172.21.15.100:0/2284371419 >> v1:172.21.15.122:6789/0 conn(0x7fed1c06f790 legacy=0x7fed1c1375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:57:04.659 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed1bfff640 10 monclient: _finish_hunting 0 2024-09-13T11:57:04.659 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed1bfff640 1 monclient: found mon.noname-b 2024-09-13T11:57:04.659 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:04.659 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed1bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:57:04.659 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.642+0000 7fed1bfff640 1 -- 172.21.15.100:0/2284371419 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed1c06fca0 con 0x7fed1c06ea50 2024-09-13T11:57:04.659 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.643+0000 7fed1bfff640 1 --2- 172.21.15.100:0/2284371419 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c06ea50 0x7fed1c0a2860 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009870 tx=0x7fed0c02a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b85243d2e07f5e40 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:04.659 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.643+0000 7fed1affd640 1 -- 172.21.15.100:0/2284371419 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c037070 con 0x7fed1c06ea50 2024-09-13T11:57:04.659 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.643+0000 7fed1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.659 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.643+0000 7fed1affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:57:04.659 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.643+0000 7fed1affd640 10 monclient: dump: 2024-09-13T11:57:04.659 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.660 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.660 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.660 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.660 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.660 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.660 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.660 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.660 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.660 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.660 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.643+0000 7fed1affd640 10 monclient: _finish_auth 0 2024-09-13T11:57:04.661 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.643+0000 7fed1affd640 10 monclient: _check_auth_tickets 2024-09-13T11:57:04.661 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.643+0000 7fed1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:04.661 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.643+0000 7fed1affd640 1 -- 172.21.15.100:0/2284371419 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed0c02adf0 con 0x7fed1c06ea50 2024-09-13T11:57:04.661 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.643+0000 7fed225f5640 5 monclient: authenticate success, global_id 24140 2024-09-13T11:57:04.661 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.643+0000 7fed225f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:57:04.661 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.643+0000 7fed1affd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:04.661 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.643+0000 7fed225f5640 10 monclient: get_monmap_and_config success 2024-09-13T11:57:04.661 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.643+0000 7fed1affd640 1 -- 172.21.15.100:0/2284371419 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c032660 con 0x7fed1c06ea50 2024-09-13T11:57:04.661 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.643+0000 7fed225f5640 4 set_mon_vals no callback set 2024-09-13T11:57:04.661 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.643+0000 7fed1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.661 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.643+0000 7fed1affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:57:04.662 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.643+0000 7fed1affd640 10 monclient: dump: 2024-09-13T11:57:04.662 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.662 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.662 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.662 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.662 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.662 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.662 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.662 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.662 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.662 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.644+0000 7fed225f5640 10 monclient: shutdown 2024-09-13T11:57:04.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.644+0000 7fed225f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:04.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.644+0000 7fed225f5640 1 -- 172.21.15.100:0/2284371419 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c06ea50 msgr2=0x7fed1c0a2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.644+0000 7fed225f5640 1 --2- 172.21.15.100:0/2284371419 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c06ea50 0x7fed1c0a2860 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009870 tx=0x7fed0c02a0a0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.644+0000 7fed225f5640 1 -- 172.21.15.100:0/2284371419 shutdown_connections 2024-09-13T11:57:04.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.644+0000 7fed225f5640 1 --2- 172.21.15.100:0/2284371419 >> v2:172.21.15.122:3300/0 conn(0x7fed1c06ee30 0x7fed1c06f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.644+0000 7fed225f5640 1 --2- 172.21.15.100:0/2284371419 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c06ea50 0x7fed1c0a2860 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.644+0000 7fed225f5640 1 -- 172.21.15.100:0/2284371419 >> 172.21.15.100:0/2284371419 conn(0x7fed1c0a68e0 msgr2=0x7fed1c0a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:04.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.644+0000 7fed225f5640 1 -- 172.21.15.100:0/2284371419 shutdown_connections 2024-09-13T11:57:04.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.644+0000 7fed225f5640 1 -- 172.21.15.100:0/2284371419 wait complete. 2024-09-13T11:57:04.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.644+0000 7fed225f5640 10 monclient: build_initial_monmap 2024-09-13T11:57:04.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.644+0000 7fed225f5640 10 monclient: monmap: 2024-09-13T11:57:04.664 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:04.664 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:57:04.664 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:57:04.664 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:57:04.664 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:04.664 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.664 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:57:04.664 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:57:04.664 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:57:04.664 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.645+0000 7fed225f5640 1 Processor -- start 2024-09-13T11:57:04.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.645+0000 7fed225f5640 1 -- start start 2024-09-13T11:57:04.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.645+0000 7fed225f5640 10 monclient: init 2024-09-13T11:57:04.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.645+0000 7fed225f5640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:04.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.645+0000 7fed225f5640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:57:04.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.645+0000 7fed225f5640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c06ea50 0x7fed1c124070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.645+0000 7fed225f5640 10 monclient(hunting): picked mon.noname-c con 0x7fed1c06ea50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:04.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.645+0000 7fed225f5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed1c06ee30 0x7fed1c1245b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.645+0000 7fed225f5640 10 monclient(hunting): picked mon.noname-b con 0x7fed1c06ee30 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:57:04.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.645+0000 7fed225f5640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed1c06f790 0x7fed1c124af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.645+0000 7fed225f5640 10 monclient(hunting): picked mon.noname-a con 0x7fed1c06f790 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:57:04.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.645+0000 7fed225f5640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.645+0000 7fed225f5640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c137e30 con 0x7fed1c06f790 2024-09-13T11:57:04.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.645+0000 7fed225f5640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.645+0000 7fed225f5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c137cb0 con 0x7fed1c06ee30 2024-09-13T11:57:04.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.645+0000 7fed225f5640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.645+0000 7fed225f5640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fed1c137fb0 con 0x7fed1c06ea50 2024-09-13T11:57:04.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.645+0000 7fed225f5640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:04.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.645+0000 7fed225f5640 10 monclient(hunting): authenticate will time out at 1005.238895s 2024-09-13T11:57:04.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.645+0000 7fed1bfff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c06ea50 0x7fed1c124070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.645+0000 7fed1bfff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c06ea50 0x7fed1c124070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:46152/0 (socket says 172.21.15.100:46152) 2024-09-13T11:57:04.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed1bfff640 1 -- 172.21.15.100:0/2097952147 learned_addr learned my addr 172.21.15.100:0/2097952147 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:04.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed1bfff640 10 monclient(hunting): get_auth_request con 0x7fed1c06ea50 auth_method 0 2024-09-13T11:57:04.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed20b92640 1 --2- 172.21.15.100:0/2097952147 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed1c06f790 0x7fed1c124af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed20b92640 10 monclient(hunting): get_auth_request con 0x7fed1c06f790 auth_method 0 2024-09-13T11:57:04.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed20b92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed20b92640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed20b92640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed20b92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed20b92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed20b92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed1b7fe640 1 --2- 172.21.15.100:0/2097952147 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed1c06ee30 0x7fed1c1245b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed1bfff640 10 monclient(hunting): handle_auth_done global_id 24143 payload 824 2024-09-13T11:57:04.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed1bfff640 1 -- 172.21.15.100:0/2097952147 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed1c06ee30 msgr2=0x7fed1c1245b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed1bfff640 1 --2- 172.21.15.100:0/2097952147 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed1c06ee30 0x7fed1c1245b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed1bfff640 1 -- 172.21.15.100:0/2097952147 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed1c06f790 msgr2=0x7fed1c124af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed1bfff640 1 --2- 172.21.15.100:0/2097952147 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed1c06f790 0x7fed1c124af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed1bfff640 10 monclient: _finish_hunting 0 2024-09-13T11:57:04.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed1bfff640 1 monclient: found mon.noname-c 2024-09-13T11:57:04.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:04.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed1bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:57:04.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed1bfff640 1 -- 172.21.15.100:0/2097952147 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed1c0a10d0 con 0x7fed1c06ea50 2024-09-13T11:57:04.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed20b92640 1 --2- 172.21.15.100:0/2097952147 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed1c06f790 0x7fed1c124af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:57:04.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed1bfff640 1 --2- 172.21.15.100:0/2097952147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c06ea50 0x7fed1c124070 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009870 tx=0x7fed0c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:04.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed18ff9640 1 -- 172.21.15.100:0/2097952147 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c041020 con 0x7fed1c06ea50 2024-09-13T11:57:04.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed18ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:57:04.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed18ff9640 10 monclient: dump: 2024-09-13T11:57:04.673 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.673 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.673 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.673 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.673 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.673 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.674 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.674 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.674 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.674 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed18ff9640 10 monclient: _finish_auth 0 2024-09-13T11:57:04.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed18ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:57:04.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:04.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed225f5640 5 monclient: authenticate success, global_id 24143 2024-09-13T11:57:04.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed18ff9640 1 -- 172.21.15.100:0/2097952147 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed0c03b610 con 0x7fed1c06ea50 2024-09-13T11:57:04.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed225f5640 10 monclient: _renew_subs 2024-09-13T11:57:04.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed225f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:57:04.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed225f5640 1 -- 172.21.15.100:0/2097952147 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed1c127830 con 0x7fed1c06ea50 2024-09-13T11:57:04.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed18ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:04.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed18ff9640 1 -- 172.21.15.100:0/2097952147 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c037040 con 0x7fed1c06ea50 2024-09-13T11:57:04.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed18ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:57:04.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed18ff9640 10 monclient: dump: 2024-09-13T11:57:04.675 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.676 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.676 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.676 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.676 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.676 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.676 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.676 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.677 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.677 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed225f5640 10 monclient: _renew_subs 2024-09-13T11:57:04.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed225f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:57:04.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed1a7fc640 4 set_mon_vals no callback set 2024-09-13T11:57:04.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.646+0000 7fed225f5640 1 -- 172.21.15.100:0/2097952147 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fed1c1282a0 con 0x7fed1c06ea50 2024-09-13T11:57:04.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.647+0000 7fed225f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:04.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.647+0000 7fed225f5640 1 --2- 172.21.15.100:0/2097952147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c1cc7d0 0x7fed1c1ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.647+0000 7fed225f5640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.647+0000 7fed225f5640 1 -- 172.21.15.100:0/2097952147 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fed1c127f50 con 0x7fed1c1cc7d0 2024-09-13T11:57:04.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.647+0000 7fed18ff9640 1 -- 172.21.15.100:0/2097952147 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fed0c044020 con 0x7fed1c06ea50 2024-09-13T11:57:04.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.647+0000 7fed1b7fe640 1 --2- 172.21.15.100:0/2097952147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c1cc7d0 0x7fed1c1ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.647+0000 7fed1b7fe640 10 monclient: get_auth_request con 0x7fed1c1cc7d0 auth_method 0 2024-09-13T11:57:04.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.647+0000 7fed1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.647+0000 7fed1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.647+0000 7fed1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.647+0000 7fed1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.647+0000 7fed1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.647+0000 7fed1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.647+0000 7fed18ff9640 1 --2- 172.21.15.100:0/2097952147 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fecfc03dfa0 0x7fecfc040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.647+0000 7fed18ff9640 1 -- 172.21.15.100:0/2097952147 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(223..223 src has 1..223) ==== 4346+0+0 (secure 0 0 0) 0x7fed0c072670 con 0x7fed1c06ea50 2024-09-13T11:57:04.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.647+0000 7fed1b7fe640 10 monclient(hunting): handle_auth_done global_id 24146 payload 824 2024-09-13T11:57:04.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.647+0000 7fed1b7fe640 1 -- 172.21.15.100:0/2097952147 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fed1c126f80 con 0x7fed1c1cc7d0 2024-09-13T11:57:04.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.647+0000 7fed20b92640 1 --2- 172.21.15.100:0/2097952147 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fecfc03dfa0 0x7fecfc040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.647+0000 7fed20b92640 10 monclient: get_auth_request con 0x7fecfc03dfa0 auth_method 0 2024-09-13T11:57:04.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.647+0000 7fed1b7fe640 1 --2- 172.21.15.100:0/2097952147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c1cc7d0 0x7fed1c1ccbb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fed1c06f660 tx=0x7fed0400eba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:04.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.648+0000 7fed18ff9640 1 -- 172.21.15.100:0/2097952147 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed04017070 con 0x7fed1c1cc7d0 2024-09-13T11:57:04.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.648+0000 7fed18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.648+0000 7fed18ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:57:04.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.648+0000 7fed18ff9640 10 monclient: dump: 2024-09-13T11:57:04.680 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.680 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.681 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.681 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.681 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.681 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.681 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.681 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.681 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.681 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.648+0000 7fed20b92640 1 --2- 172.21.15.100:0/2097952147 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fecfc03dfa0 0x7fecfc040460 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fed1c0a9750 tx=0x7fed14003d10 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:04.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.648+0000 7fed18ff9640 1 -- 172.21.15.100:0/2097952147 <== mon.2 v2:172.21.15.100:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fed0400c740 con 0x7fed1c1cc7d0 2024-09-13T11:57:04.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.648+0000 7fed18ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:04.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.648+0000 7fed18ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:57:04.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.648+0000 7fed18ff9640 1 -- 172.21.15.100:0/2097952147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c1cc7d0 msgr2=0x7fed1c1ccbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.648+0000 7fed18ff9640 1 --2- 172.21.15.100:0/2097952147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c1cc7d0 0x7fed1c1ccbb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fed1c06f660 tx=0x7fed0400eba0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.648+0000 7fed18ff9640 1 -- 172.21.15.100:0/2097952147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c1cc7d0 msgr2=0x7fed1c1ccbb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:57:04.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.648+0000 7fed18ff9640 1 --2- 172.21.15.100:0/2097952147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c1cc7d0 0x7fed1c1ccbb0 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fed1c06f660 tx=0x7fed0400eba0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.654+0000 7fed225f5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:57:04.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.654+0000 7fed225f5640 1 --2- 172.21.15.100:0/2097952147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c1cd0f0 0x7fed1c120430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.654+0000 7fed225f5640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.654+0000 7fed225f5640 1 -- 172.21.15.100:0/2097952147 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fed1c125a90 con 0x7fed1c1cd0f0 2024-09-13T11:57:04.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.654+0000 7fed1b7fe640 1 --2- 172.21.15.100:0/2097952147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c1cd0f0 0x7fed1c120430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.654+0000 7fed1b7fe640 10 monclient: get_auth_request con 0x7fed1c1cd0f0 auth_method 0 2024-09-13T11:57:04.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.654+0000 7fed1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.654+0000 7fed1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.654+0000 7fed1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.654+0000 7fed1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.654+0000 7fed1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.654+0000 7fed1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.654+0000 7fed1b7fe640 10 monclient(hunting): handle_auth_done global_id 24149 payload 824 2024-09-13T11:57:04.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.654+0000 7fed1b7fe640 1 -- 172.21.15.100:0/2097952147 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fed1c0a2860 con 0x7fed1c1cd0f0 2024-09-13T11:57:04.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.654+0000 7fed1b7fe640 1 --2- 172.21.15.100:0/2097952147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c1cd0f0 0x7fed1c120430 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fed1c06f660 tx=0x7fed040046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:04.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.655+0000 7fed18ff9640 1 -- 172.21.15.100:0/2097952147 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0400f040 con 0x7fed1c1cd0f0 2024-09-13T11:57:04.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.655+0000 7fed18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.655+0000 7fed18ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:57:04.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.655+0000 7fed18ff9640 10 monclient: dump: 2024-09-13T11:57:04.686 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.686 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.686 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.686 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.686 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.686 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.686 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.686 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.686 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.686 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.655+0000 7fed18ff9640 1 -- 172.21.15.100:0/2097952147 <== mon.2 v2:172.21.15.100:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7fed04003820 con 0x7fed1c1cd0f0 2024-09-13T11:57:04.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.655+0000 7fed18ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:57:04.687 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.655+0000 7fed18ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:57:04.687 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.655+0000 7fed18ff9640 1 -- 172.21.15.100:0/2097952147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c1cd0f0 msgr2=0x7fed1c120430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.687 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.655+0000 7fed18ff9640 1 --2- 172.21.15.100:0/2097952147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c1cd0f0 0x7fed1c120430 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fed1c06f660 tx=0x7fed040046c0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.687 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.655+0000 7fed18ff9640 1 -- 172.21.15.100:0/2097952147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c1cd0f0 msgr2=0x7fed1c120430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:57:04.687 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.655+0000 7fed18ff9640 1 --2- 172.21.15.100:0/2097952147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c1cd0f0 0x7fed1c120430 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fed1c06f660 tx=0x7fed040046c0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.687 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:57:04.687 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:57:04.687 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:57:04.687 INFO:teuthology.orchestra.run.smithi100.stdout: "state": "peon", 2024-09-13T11:57:04.688 INFO:teuthology.orchestra.run.smithi100.stdout: "election_epoch": 22, 2024-09-13T11:57:04.688 INFO:teuthology.orchestra.run.smithi100.stdout: "uptime": 17632, 2024-09-13T11:57:04.688 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum": [ 2024-09-13T11:57:04.688 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-13T11:57:04.688 INFO:teuthology.orchestra.run.smithi100.stdout: 1, 2024-09-13T11:57:04.688 INFO:teuthology.orchestra.run.smithi100.stdout: 2 2024-09-13T11:57:04.688 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:04.688 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_age": 6, 2024-09-13T11:57:04.688 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:57:04.688 INFO:teuthology.orchestra.run.smithi100.stdout: "required_con": "2450521708007866372", 2024-09-13T11:57:04.688 INFO:teuthology.orchestra.run.smithi100.stdout: "required_mon": [ 2024-09-13T11:57:04.689 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:57:04.689 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:57:04.689 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:57:04.689 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:57:04.689 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:57:04.689 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:57:04.689 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:57:04.689 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:57:04.689 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:57:04.689 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:57:04.689 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:57:04.690 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:04.690 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:57:04.690 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_mon": [ 2024-09-13T11:57:04.690 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:57:04.690 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:57:04.690 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:57:04.690 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:57:04.690 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:57:04.690 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:57:04.690 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:57:04.690 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:57:04.690 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:57:04.691 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:57:04.691 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:57:04.691 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:04.691 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.691 INFO:teuthology.orchestra.run.smithi100.stdout: "outside_quorum": [], 2024-09-13T11:57:04.691 INFO:teuthology.orchestra.run.smithi100.stdout: "extra_probe_peers": [], 2024-09-13T11:57:04.691 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_provider": [], 2024-09-13T11:57:04.691 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": { 2024-09-13T11:57:04.691 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-09-13T11:57:04.691 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "0b1f4b4d-fad4-4dd3-86ac-a286d80846c0", 2024-09-13T11:57:04.691 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:57:04.691 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:57:04.692 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release": 19, 2024-09-13T11:57:04.692 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release_name": "squid", 2024-09-13T11:57:04.692 INFO:teuthology.orchestra.run.smithi100.stdout: "election_strategy": 1, 2024-09-13T11:57:04.692 INFO:teuthology.orchestra.run.smithi100.stdout: "disallowed_leaders: ": "", 2024-09-13T11:57:04.692 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false, 2024-09-13T11:57:04.692 INFO:teuthology.orchestra.run.smithi100.stdout: "tiebreaker_mon": "", 2024-09-13T11:57:04.692 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_ranks: ": "", 2024-09-13T11:57:04.692 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:57:04.692 INFO:teuthology.orchestra.run.smithi100.stdout: "persistent": [ 2024-09-13T11:57:04.692 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:57:04.692 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:57:04.693 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:57:04.693 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:57:04.693 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:57:04.693 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:57:04.693 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:57:04.693 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:57:04.693 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:57:04.693 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:57:04.693 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:57:04.693 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:04.693 INFO:teuthology.orchestra.run.smithi100.stdout: "optional": [] 2024-09-13T11:57:04.693 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.694 INFO:teuthology.orchestra.run.smithi100.stdout: "mons": [ 2024-09-13T11:57:04.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.694 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:57:04.694 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:57:04.694 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:57:04.694 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:57:04.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.694 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:57:04.694 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3300", 2024-09-13T11:57:04.694 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:04.694 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.695 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.695 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:57:04.695 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789", 2024-09-13T11:57:04.695 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:04.695 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:04.695 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:04.695 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.695 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789/0", 2024-09-13T11:57:04.695 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-13T11:57:04.695 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:57:04.695 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:57:04.696 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:57:04.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.696 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:57:04.696 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:57:04.696 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:57:04.696 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:57:04.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.696 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:57:04.696 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:3300", 2024-09-13T11:57:04.696 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:04.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.697 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:57:04.697 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789", 2024-09-13T11:57:04.697 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:04.697 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:04.697 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:04.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.697 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789/0", 2024-09-13T11:57:04.697 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.122:6789/0", 2024-09-13T11:57:04.697 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:57:04.697 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:57:04.698 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:57:04.698 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.698 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:57:04.698 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:57:04.698 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:57:04.698 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:57:04.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.699 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:57:04.699 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3301", 2024-09-13T11:57:04.699 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:04.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.699 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:57:04.699 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790", 2024-09-13T11:57:04.699 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:04.699 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:04.699 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:04.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.700 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790/0", 2024-09-13T11:57:04.700 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-13T11:57:04.700 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:57:04.700 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:57:04.700 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:57:04.700 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:04.700 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:04.700 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.700 INFO:teuthology.orchestra.run.smithi100.stdout: "feature_map": { 2024-09-13T11:57:04.700 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": [ 2024-09-13T11:57:04.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.701 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:04.701 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:04.701 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:57:04.701 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:04.701 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:04.701 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": [ 2024-09-13T11:57:04.701 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.701 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:04.702 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:04.702 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:57:04.702 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:04.702 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:04.702 INFO:teuthology.orchestra.run.smithi100.stdout: "client": [ 2024-09-13T11:57:04.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.702 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:04.702 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:04.702 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 3 2024-09-13T11:57:04.702 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:04.702 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:04.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.703 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false 2024-09-13T11:57:04.703 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:57:04.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.655+0000 7fed225f5640 1 -- 172.21.15.100:0/2097952147 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fecfc03dfa0 msgr2=0x7fecfc040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.655+0000 7fed225f5640 1 --2- 172.21.15.100:0/2097952147 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fecfc03dfa0 0x7fecfc040460 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fed1c0a9750 tx=0x7fed14003d10 comp rx=0 tx=0).stop 2024-09-13T11:57:04.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.655+0000 7fed225f5640 10 monclient: shutdown 2024-09-13T11:57:04.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.655+0000 7fed225f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:04.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.655+0000 7fed225f5640 1 -- 172.21.15.100:0/2097952147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c06ea50 msgr2=0x7fed1c124070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.655+0000 7fed225f5640 1 --2- 172.21.15.100:0/2097952147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed1c06ea50 0x7fed1c124070 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009870 tx=0x7fed0c0046c0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.655+0000 7fed20b92640 1 -- 172.21.15.100:0/2097952147 reap_dead start 2024-09-13T11:57:04.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.655+0000 7fed20b92640 1 -- 172.21.15.100:0/2097952147 reap_dead start 2024-09-13T11:57:04.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.655+0000 7fed225f5640 1 -- 172.21.15.100:0/2097952147 shutdown_connections 2024-09-13T11:57:04.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.655+0000 7fed225f5640 1 -- 172.21.15.100:0/2097952147 >> 172.21.15.100:0/2097952147 conn(0x7fed1c0a68e0 msgr2=0x7fed1c0a8500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:04.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.655+0000 7fed225f5640 1 -- 172.21.15.100:0/2097952147 shutdown_connections 2024-09-13T11:57:04.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.655+0000 7fed225f5640 1 -- 172.21.15.100:0/2097952147 wait complete. 2024-09-13T11:57:04.706 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T11:57:04.706 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-13T11:57:04.707 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-13T11:57:04.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.804+0000 7f1d45b89640 10 monclient: get_monmap_and_config 2024-09-13T11:57:04.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.804+0000 7f1d45b89640 10 monclient: build_initial_monmap 2024-09-13T11:57:04.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.804+0000 7f1d45b89640 10 monclient: monmap: 2024-09-13T11:57:04.806 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:04.806 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.806 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:57:04.805016+0000 2024-09-13T11:57:04.806 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:57:04.805016+0000 2024-09-13T11:57:04.806 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:04.806 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.806 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:57:04.806 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:57:04.806 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:57:04.806 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:57:04.806 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:57:04.806 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.804+0000 7f1d45b89640 1 Processor -- start 2024-09-13T11:57:04.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.804+0000 7f1d45b89640 1 -- start start 2024-09-13T11:57:04.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.804+0000 7f1d45b89640 10 monclient: init 2024-09-13T11:57:04.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d45b89640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:04.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d45b89640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-13T11:57:04.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d45b89640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1d4006ea50 addr v1:172.21.15.100:6789/0 2024-09-13T11:57:04.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d45b89640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1d400a2860 0x7f1d4006f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d45b89640 10 monclient(hunting): picked mon.noname-c con 0x7f1d400a2860 addr v2:172.21.15.122:3300/0 2024-09-13T11:57:04.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d45b89640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d4013c9f0 0x7f1d4013ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d45b89640 10 monclient(hunting): picked mon.noname-b con 0x7f1d4013c9f0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:04.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d45b89640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d4006fa80 con 0x7f1d4006ea50 2024-09-13T11:57:04.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d45b89640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d45b89640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f1d400a2cb0 con 0x7f1d400a2860 2024-09-13T11:57:04.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d45b89640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d45b89640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f1d400a15e0 con 0x7f1d4013c9f0 2024-09-13T11:57:04.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d45b89640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:04.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d45b89640 10 monclient(hunting): authenticate will time out at 1005.398482s 2024-09-13T11:57:04.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3f7fe640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f1d4006ea50 0x7f1d4006ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:47478/0 (socket says 172.21.15.100:47478) 2024-09-13T11:57:04.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3f7fe640 1 -- 172.21.15.100:0/1192473859 learned_addr learned my addr 172.21.15.100:0/1192473859 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:57:04.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3ffff640 1 --2- 172.21.15.100:0/1192473859 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d4013c9f0 0x7f1d4013ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3ffff640 10 monclient(hunting): get_auth_request con 0x7f1d4013c9f0 auth_method 0 2024-09-13T11:57:04.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3effd640 1 --2- 172.21.15.100:0/1192473859 >> v2:172.21.15.122:3300/0 conn(0x7f1d400a2860 0x7f1d4006f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3e7fc640 1 -- 172.21.15.100:0/1192473859 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3112433731 0 0) 0x7f1d30001490 con 0x7f1d4006ea50 2024-09-13T11:57:04.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3e7fc640 10 monclient(hunting): _negotiate 2024-09-13T11:57:04.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3e7fc640 10 monclient(hunting): my global_id is 14916 2024-09-13T11:57:04.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3e7fc640 1 -- 172.21.15.100:0/1192473859 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d28003520 con 0x7f1d4006ea50 2024-09-13T11:57:04.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3ffff640 10 monclient(hunting): handle_auth_done global_id 24152 payload 274 2024-09-13T11:57:04.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3ffff640 1 -- 172.21.15.100:0/1192473859 >> v2:172.21.15.122:3300/0 conn(0x7f1d400a2860 msgr2=0x7f1d4006f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:57:04.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3ffff640 1 --2- 172.21.15.100:0/1192473859 >> v2:172.21.15.122:3300/0 conn(0x7f1d400a2860 0x7f1d4006f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3ffff640 1 -- 172.21.15.100:0/1192473859 >> v1:172.21.15.100:6789/0 conn(0x7f1d4006ea50 legacy=0x7f1d4006ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3ffff640 10 monclient: _finish_hunting 0 2024-09-13T11:57:04.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3ffff640 1 monclient: found mon.noname-b 2024-09-13T11:57:04.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:04.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:57:04.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.805+0000 7f1d3ffff640 1 -- 172.21.15.100:0/1192473859 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d400a1870 con 0x7f1d4013c9f0 2024-09-13T11:57:04.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.806+0000 7f1d3ffff640 1 --2- 172.21.15.100:0/1192473859 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d4013c9f0 0x7f1d4013ede0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1d380097a0 tx=0x7f1d3802a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81d73b4fab932435 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:04.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.806+0000 7f1d3e7fc640 1 -- 172.21.15.100:0/1192473859 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d38037070 con 0x7f1d4013c9f0 2024-09-13T11:57:04.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.806+0000 7f1d3e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.806+0000 7f1d3e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:57:04.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.806+0000 7f1d3e7fc640 10 monclient: dump: 2024-09-13T11:57:04.821 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.821 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.821 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.821 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.821 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.821 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.821 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.821 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.821 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.821 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.806+0000 7f1d3e7fc640 10 monclient: _finish_auth 0 2024-09-13T11:57:04.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.806+0000 7f1d3e7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:57:04.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.806+0000 7f1d3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:04.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.806+0000 7f1d45b89640 5 monclient: authenticate success, global_id 24152 2024-09-13T11:57:04.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.806+0000 7f1d3e7fc640 1 -- 172.21.15.100:0/1192473859 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d3802adb0 con 0x7f1d4013c9f0 2024-09-13T11:57:04.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.806+0000 7f1d3e7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:04.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.806+0000 7f1d45b89640 10 monclient: get_monmap_and_config success 2024-09-13T11:57:04.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.806+0000 7f1d45b89640 4 set_mon_vals no callback set 2024-09-13T11:57:04.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.806+0000 7f1d3e7fc640 1 -- 172.21.15.100:0/1192473859 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d38032660 con 0x7f1d4013c9f0 2024-09-13T11:57:04.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.806+0000 7f1d3e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.806+0000 7f1d3e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:57:04.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.806+0000 7f1d3e7fc640 10 monclient: dump: 2024-09-13T11:57:04.822 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.822 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.823 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.823 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.823 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.823 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.823 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.823 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.823 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.823 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.807+0000 7f1d45b89640 10 monclient: shutdown 2024-09-13T11:57:04.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.807+0000 7f1d45b89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:04.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.807+0000 7f1d45b89640 1 -- 172.21.15.100:0/1192473859 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d4013c9f0 msgr2=0x7f1d4013ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.807+0000 7f1d45b89640 1 --2- 172.21.15.100:0/1192473859 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d4013c9f0 0x7f1d4013ede0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1d380097a0 tx=0x7f1d3802a060 comp rx=0 tx=0).stop 2024-09-13T11:57:04.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.807+0000 7f1d45b89640 1 -- 172.21.15.100:0/1192473859 shutdown_connections 2024-09-13T11:57:04.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.807+0000 7f1d45b89640 1 --2- 172.21.15.100:0/1192473859 >> v2:172.21.15.122:3300/0 conn(0x7f1d400a2860 0x7f1d4006f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.807+0000 7f1d45b89640 1 --2- 172.21.15.100:0/1192473859 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d4013c9f0 0x7f1d4013ede0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.807+0000 7f1d45b89640 1 -- 172.21.15.100:0/1192473859 >> 172.21.15.100:0/1192473859 conn(0x7f1d400a68e0 msgr2=0x7f1d400a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:04.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.807+0000 7f1d45b89640 1 -- 172.21.15.100:0/1192473859 shutdown_connections 2024-09-13T11:57:04.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.807+0000 7f1d45b89640 1 -- 172.21.15.100:0/1192473859 wait complete. 2024-09-13T11:57:04.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.807+0000 7f1d45b89640 10 monclient: build_initial_monmap 2024-09-13T11:57:04.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.807+0000 7f1d45b89640 10 monclient: monmap: 2024-09-13T11:57:04.824 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:04.824 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:57:04.824 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:57:04.824 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:57:04.824 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:04.824 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.824 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:57:04.825 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:57:04.825 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:57:04.825 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d45b89640 1 Processor -- start 2024-09-13T11:57:04.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d45b89640 1 -- start start 2024-09-13T11:57:04.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d45b89640 10 monclient: init 2024-09-13T11:57:04.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d45b89640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:04.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d45b89640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:57:04.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d45b89640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d4006ea50 0x7f1d4009f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d45b89640 10 monclient(hunting): picked mon.noname-b con 0x7f1d4006ea50 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:57:04.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d45b89640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d400a2860 0x7f1d40120c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d45b89640 10 monclient(hunting): picked mon.noname-a con 0x7f1d400a2860 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:57:04.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d45b89640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d4013c9f0 0x7f1d401379b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d45b89640 10 monclient(hunting): picked mon.noname-c con 0x7f1d4013c9f0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:04.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d45b89640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d45b89640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f1d400a5be0 con 0x7f1d400a2860 2024-09-13T11:57:04.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d45b89640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d45b89640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1d400a5a60 con 0x7f1d4006ea50 2024-09-13T11:57:04.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d45b89640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d45b89640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f1d4013f320 con 0x7f1d4013c9f0 2024-09-13T11:57:04.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d45b89640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:04.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d45b89640 10 monclient(hunting): authenticate will time out at 1005.401835s 2024-09-13T11:57:04.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d3effd640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d400a2860 0x7f1d40120c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d3ffff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d4013c9f0 0x7f1d401379b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d3effd640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d400a2860 0x7f1d40120c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55480/0 (socket says 172.21.15.100:55480) 2024-09-13T11:57:04.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d3effd640 1 -- 172.21.15.100:0/3995125134 learned_addr learned my addr 172.21.15.100:0/3995125134 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:04.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d3effd640 10 monclient(hunting): get_auth_request con 0x7f1d400a2860 auth_method 0 2024-09-13T11:57:04.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d3effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d3ffff640 10 monclient(hunting): get_auth_request con 0x7f1d4013c9f0 auth_method 0 2024-09-13T11:57:04.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.808+0000 7f1d3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d3f7fe640 1 --2- 172.21.15.100:0/3995125134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d4006ea50 0x7f1d4009f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d3ffff640 10 monclient(hunting): handle_auth_done global_id 24155 payload 824 2024-09-13T11:57:04.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d3ffff640 1 -- 172.21.15.100:0/3995125134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d4006ea50 msgr2=0x7f1d4009f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d3ffff640 1 --2- 172.21.15.100:0/3995125134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d4006ea50 0x7f1d4009f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d3ffff640 1 -- 172.21.15.100:0/3995125134 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d400a2860 msgr2=0x7f1d40120c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d3ffff640 1 --2- 172.21.15.100:0/3995125134 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d400a2860 0x7f1d40120c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d3ffff640 10 monclient: _finish_hunting 0 2024-09-13T11:57:04.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d3ffff640 1 monclient: found mon.noname-c 2024-09-13T11:57:04.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:04.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d3ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:57:04.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d3ffff640 1 -- 172.21.15.100:0/3995125134 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d400a10d0 con 0x7f1d4013c9f0 2024-09-13T11:57:04.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d3f7fe640 10 monclient: get_auth_request con 0x7f1d4006ea50 auth_method 0 2024-09-13T11:57:04.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d3f7fe640 1 --2- 172.21.15.100:0/3995125134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d4006ea50 0x7f1d4009f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:57:04.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d3effd640 1 --2- 172.21.15.100:0/3995125134 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d400a2860 0x7f1d40120c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:57:04.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d3ffff640 1 --2- 172.21.15.100:0/3995125134 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d4013c9f0 0x7f1d401379b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1d400a7960 tx=0x7f1d380046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:04.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d27fff640 1 -- 172.21.15.100:0/3995125134 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d38041070 con 0x7f1d4013c9f0 2024-09-13T11:57:04.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d27fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:57:04.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d27fff640 10 monclient: dump: 2024-09-13T11:57:04.829 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.829 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.829 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.829 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.829 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.829 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.829 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.829 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.830 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.830 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d27fff640 10 monclient: _finish_auth 0 2024-09-13T11:57:04.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d27fff640 10 monclient: _check_auth_tickets 2024-09-13T11:57:04.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:04.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d45b89640 5 monclient: authenticate success, global_id 24155 2024-09-13T11:57:04.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d27fff640 1 -- 172.21.15.100:0/3995125134 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d3803c460 con 0x7f1d4013c9f0 2024-09-13T11:57:04.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d45b89640 10 monclient: _renew_subs 2024-09-13T11:57:04.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d45b89640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:57:04.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d45b89640 1 -- 172.21.15.100:0/3995125134 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d40121e80 con 0x7f1d4013c9f0 2024-09-13T11:57:04.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d27fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:04.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d27fff640 1 -- 172.21.15.100:0/3995125134 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d38037040 con 0x7f1d4013c9f0 2024-09-13T11:57:04.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d27fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:57:04.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d27fff640 10 monclient: dump: 2024-09-13T11:57:04.831 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.831 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.831 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.831 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.831 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.831 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.831 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.831 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.831 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.831 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d45b89640 10 monclient: _renew_subs 2024-09-13T11:57:04.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d45b89640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:57:04.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d3dffb640 4 set_mon_vals no callback set 2024-09-13T11:57:04.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.809+0000 7f1d45b89640 1 -- 172.21.15.100:0/3995125134 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d40120430 con 0x7f1d4013c9f0 2024-09-13T11:57:04.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.810+0000 7f1d45b89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:04.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.810+0000 7f1d45b89640 1 --2- 172.21.15.100:0/3995125134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d40137ef0 0x7f1d401cc770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.810+0000 7f1d45b89640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.810+0000 7f1d45b89640 1 -- 172.21.15.100:0/3995125134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1d400a0970 con 0x7f1d40137ef0 2024-09-13T11:57:04.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.810+0000 7f1d27fff640 1 -- 172.21.15.100:0/3995125134 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f1d38044070 con 0x7f1d4013c9f0 2024-09-13T11:57:04.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.810+0000 7f1d3f7fe640 1 --2- 172.21.15.100:0/3995125134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d40137ef0 0x7f1d401cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.810+0000 7f1d3f7fe640 10 monclient: get_auth_request con 0x7f1d40137ef0 auth_method 0 2024-09-13T11:57:04.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.810+0000 7f1d3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.810+0000 7f1d3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.832 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.810+0000 7f1d3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.810+0000 7f1d27fff640 1 --2- 172.21.15.100:0/3995125134 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f1d2c03dfd0 0x7f1d2c040490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.810+0000 7f1d3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.810+0000 7f1d3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.810+0000 7f1d3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.810+0000 7f1d27fff640 1 -- 172.21.15.100:0/3995125134 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(223..223 src has 1..223) ==== 4346+0+0 (secure 0 0 0) 0x7f1d38072be0 con 0x7f1d4013c9f0 2024-09-13T11:57:04.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.810+0000 7f1d3effd640 1 --2- 172.21.15.100:0/3995125134 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f1d2c03dfd0 0x7f1d2c040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.810+0000 7f1d3effd640 10 monclient: get_auth_request con 0x7f1d2c03dfd0 auth_method 0 2024-09-13T11:57:04.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.810+0000 7f1d3f7fe640 10 monclient(hunting): handle_auth_done global_id 5683 payload 824 2024-09-13T11:57:04.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.810+0000 7f1d3f7fe640 1 -- 172.21.15.100:0/3995125134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1d4006ee30 con 0x7f1d40137ef0 2024-09-13T11:57:04.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.811+0000 7f1d3f7fe640 1 --2- 172.21.15.100:0/3995125134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d40137ef0 0x7f1d401cc770 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1d30005c00 tx=0x7f1d3000b4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:04.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.811+0000 7f1d3effd640 1 --2- 172.21.15.100:0/3995125134 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f1d2c03dfd0 0x7f1d2c040490 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1d400a8ab0 tx=0x7f1d34008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:04.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.811+0000 7f1d27fff640 1 -- 172.21.15.100:0/3995125134 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d3000bd30 con 0x7f1d40137ef0 2024-09-13T11:57:04.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.811+0000 7f1d27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.811+0000 7f1d27fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:57:04.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.811+0000 7f1d27fff640 10 monclient: dump: 2024-09-13T11:57:04.834 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.834 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.834 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.834 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.834 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.834 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.834 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.834 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.834 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.834 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.811+0000 7f1d27fff640 1 -- 172.21.15.100:0/3995125134 <== mon.1 v2:172.21.15.122:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1d30017070 con 0x7f1d40137ef0 2024-09-13T11:57:04.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.811+0000 7f1d27fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:04.834 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.811+0000 7f1d27fff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:57:04.835 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.811+0000 7f1d27fff640 1 -- 172.21.15.100:0/3995125134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d40137ef0 msgr2=0x7f1d401cc770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.835 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.811+0000 7f1d27fff640 1 --2- 172.21.15.100:0/3995125134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d40137ef0 0x7f1d401cc770 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1d30005c00 tx=0x7f1d3000b4a0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.835 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.811+0000 7f1d27fff640 1 -- 172.21.15.100:0/3995125134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d40137ef0 msgr2=0x7f1d401cc770 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:57:04.835 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.811+0000 7f1d27fff640 1 --2- 172.21.15.100:0/3995125134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d40137ef0 0x7f1d401cc770 secure :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1d30005c00 tx=0x7f1d3000b4a0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.817+0000 7f1d45b89640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:57:04.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.817+0000 7f1d45b89640 1 --2- 172.21.15.100:0/3995125134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d401cccb0 0x7f1d401cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.817+0000 7f1d45b89640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.817+0000 7f1d45b89640 1 -- 172.21.15.100:0/3995125134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1d401cd600 con 0x7f1d401cccb0 2024-09-13T11:57:04.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.818+0000 7f1d3f7fe640 1 --2- 172.21.15.100:0/3995125134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d401cccb0 0x7f1d401cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.818+0000 7f1d3f7fe640 10 monclient: get_auth_request con 0x7f1d401cccb0 auth_method 0 2024-09-13T11:57:04.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.818+0000 7f1d3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.818+0000 7f1d3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.818+0000 7f1d3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.818+0000 7f1d3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.818+0000 7f1d3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.818+0000 7f1d3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.818+0000 7f1d3f7fe640 10 monclient(hunting): handle_auth_done global_id 5686 payload 824 2024-09-13T11:57:04.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.818+0000 7f1d3f7fe640 1 -- 172.21.15.100:0/3995125134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1d40069100 con 0x7f1d401cccb0 2024-09-13T11:57:04.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.818+0000 7f1d3f7fe640 1 --2- 172.21.15.100:0/3995125134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d401cccb0 0x7f1d401cd0c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1d30009010 tx=0x7f1d30005b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:04.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.818+0000 7f1d27fff640 1 -- 172.21.15.100:0/3995125134 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d300126e0 con 0x7f1d401cccb0 2024-09-13T11:57:04.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.818+0000 7f1d27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.818+0000 7f1d27fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:57:04.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.818+0000 7f1d27fff640 10 monclient: dump: 2024-09-13T11:57:04.838 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.838 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.838 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.839 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.839 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.839 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.839 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.839 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.839 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.839 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.818+0000 7f1d27fff640 1 -- 172.21.15.100:0/3995125134 <== mon.1 v2:172.21.15.122:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f1d300126e0 con 0x7f1d401cccb0 2024-09-13T11:57:04.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.818+0000 7f1d27fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:57:04.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.818+0000 7f1d27fff640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:57:04.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.818+0000 7f1d27fff640 1 -- 172.21.15.100:0/3995125134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d401cccb0 msgr2=0x7f1d401cd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.818+0000 7f1d27fff640 1 --2- 172.21.15.100:0/3995125134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d401cccb0 0x7f1d401cd0c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1d30009010 tx=0x7f1d30005b70 comp rx=0 tx=0).stop 2024-09-13T11:57:04.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.819+0000 7f1d27fff640 1 -- 172.21.15.100:0/3995125134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d401cccb0 msgr2=0x7f1d401cd0c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:57:04.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.819+0000 7f1d27fff640 1 --2- 172.21.15.100:0/3995125134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d401cccb0 0x7f1d401cd0c0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.840 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:57:04.840 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:57:04.840 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:57:04.840 INFO:teuthology.orchestra.run.smithi100.stdout: "state": "peon", 2024-09-13T11:57:04.840 INFO:teuthology.orchestra.run.smithi100.stdout: "election_epoch": 22, 2024-09-13T11:57:04.840 INFO:teuthology.orchestra.run.smithi100.stdout: "uptime": 263849, 2024-09-13T11:57:04.840 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum": [ 2024-09-13T11:57:04.841 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-13T11:57:04.841 INFO:teuthology.orchestra.run.smithi100.stdout: 1, 2024-09-13T11:57:04.841 INFO:teuthology.orchestra.run.smithi100.stdout: 2 2024-09-13T11:57:04.841 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:04.841 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_age": 7, 2024-09-13T11:57:04.841 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:57:04.841 INFO:teuthology.orchestra.run.smithi100.stdout: "required_con": "2450521708007866372", 2024-09-13T11:57:04.841 INFO:teuthology.orchestra.run.smithi100.stdout: "required_mon": [ 2024-09-13T11:57:04.841 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:57:04.841 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:57:04.841 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:57:04.842 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:57:04.842 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:57:04.842 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:57:04.842 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:57:04.842 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:57:04.842 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:57:04.842 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:57:04.842 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:57:04.843 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:04.843 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:57:04.843 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_mon": [ 2024-09-13T11:57:04.843 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:57:04.843 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:57:04.843 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:57:04.843 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:57:04.843 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:57:04.843 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:57:04.843 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:57:04.844 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:57:04.844 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:57:04.844 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:57:04.844 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:57:04.844 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:04.844 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.844 INFO:teuthology.orchestra.run.smithi100.stdout: "outside_quorum": [], 2024-09-13T11:57:04.844 INFO:teuthology.orchestra.run.smithi100.stdout: "extra_probe_peers": [], 2024-09-13T11:57:04.844 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_provider": [], 2024-09-13T11:57:04.844 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": { 2024-09-13T11:57:04.844 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-09-13T11:57:04.844 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "0b1f4b4d-fad4-4dd3-86ac-a286d80846c0", 2024-09-13T11:57:04.844 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:57:04.845 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:57:04.845 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release": 19, 2024-09-13T11:57:04.845 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release_name": "squid", 2024-09-13T11:57:04.845 INFO:teuthology.orchestra.run.smithi100.stdout: "election_strategy": 1, 2024-09-13T11:57:04.845 INFO:teuthology.orchestra.run.smithi100.stdout: "disallowed_leaders: ": "", 2024-09-13T11:57:04.845 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false, 2024-09-13T11:57:04.845 INFO:teuthology.orchestra.run.smithi100.stdout: "tiebreaker_mon": "", 2024-09-13T11:57:04.845 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_ranks: ": "", 2024-09-13T11:57:04.845 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:57:04.846 INFO:teuthology.orchestra.run.smithi100.stdout: "persistent": [ 2024-09-13T11:57:04.846 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:57:04.846 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:57:04.846 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:57:04.846 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:57:04.846 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:57:04.846 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:57:04.846 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:57:04.846 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:57:04.846 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:57:04.846 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:57:04.846 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:57:04.847 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:04.847 INFO:teuthology.orchestra.run.smithi100.stdout: "optional": [] 2024-09-13T11:57:04.847 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.847 INFO:teuthology.orchestra.run.smithi100.stdout: "mons": [ 2024-09-13T11:57:04.847 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.847 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:57:04.847 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:57:04.847 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:57:04.847 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:57:04.847 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.848 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:57:04.848 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3300", 2024-09-13T11:57:04.848 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:04.848 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.848 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.848 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:57:04.848 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789", 2024-09-13T11:57:04.848 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:04.848 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:04.848 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:04.849 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.849 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789/0", 2024-09-13T11:57:04.849 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-13T11:57:04.849 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:57:04.849 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:57:04.849 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:57:04.849 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.849 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.849 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:57:04.849 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:57:04.849 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:57:04.850 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:57:04.850 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.850 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:57:04.850 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:3300", 2024-09-13T11:57:04.850 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:04.850 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.850 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.850 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:57:04.850 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789", 2024-09-13T11:57:04.850 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:04.851 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:04.851 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:04.851 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.851 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789/0", 2024-09-13T11:57:04.851 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.122:6789/0", 2024-09-13T11:57:04.851 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:57:04.851 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:57:04.851 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:57:04.851 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.852 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.852 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:57:04.852 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:57:04.852 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:57:04.852 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:57:04.852 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.852 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:57:04.852 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3301", 2024-09-13T11:57:04.852 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:04.853 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.853 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.853 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:57:04.853 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790", 2024-09-13T11:57:04.853 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:04.853 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:04.853 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:04.853 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.853 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790/0", 2024-09-13T11:57:04.853 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-13T11:57:04.853 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:57:04.853 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:57:04.853 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:57:04.853 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:04.854 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:04.854 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.854 INFO:teuthology.orchestra.run.smithi100.stdout: "feature_map": { 2024-09-13T11:57:04.854 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": [ 2024-09-13T11:57:04.854 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.854 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:04.854 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:04.854 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:57:04.854 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:04.854 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:04.854 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": [ 2024-09-13T11:57:04.854 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.854 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:04.855 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:04.855 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 3 2024-09-13T11:57:04.855 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:04.855 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:04.855 INFO:teuthology.orchestra.run.smithi100.stdout: "client": [ 2024-09-13T11:57:04.855 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.855 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:04.855 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:04.855 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 5 2024-09-13T11:57:04.855 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:04.855 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:04.855 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": [ 2024-09-13T11:57:04.855 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:04.855 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:04.856 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:04.856 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:57:04.856 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:04.856 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:04.856 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:04.856 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false 2024-09-13T11:57:04.856 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:57:04.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.819+0000 7f1d45b89640 1 -- 172.21.15.100:0/3995125134 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f1d2c03dfd0 msgr2=0x7f1d2c040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.819+0000 7f1d45b89640 1 --2- 172.21.15.100:0/3995125134 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f1d2c03dfd0 0x7f1d2c040490 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1d400a8ab0 tx=0x7f1d34008040 comp rx=0 tx=0).stop 2024-09-13T11:57:04.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.819+0000 7f1d45b89640 10 monclient: shutdown 2024-09-13T11:57:04.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.819+0000 7f1d45b89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:04.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.819+0000 7f1d45b89640 1 -- 172.21.15.100:0/3995125134 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d4013c9f0 msgr2=0x7f1d401379b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.819+0000 7f1d3ffff640 1 -- 172.21.15.100:0/3995125134 reap_dead start 2024-09-13T11:57:04.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.819+0000 7f1d45b89640 1 --2- 172.21.15.100:0/3995125134 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d4013c9f0 0x7f1d401379b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1d400a7960 tx=0x7f1d380046c0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.819+0000 7f1d45b89640 1 -- 172.21.15.100:0/3995125134 shutdown_connections 2024-09-13T11:57:04.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.819+0000 7f1d45b89640 1 --2- 172.21.15.100:0/3995125134 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d4013c9f0 0x7f1d401379b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.819+0000 7f1d45b89640 1 -- 172.21.15.100:0/3995125134 >> 172.21.15.100:0/3995125134 conn(0x7f1d400a68e0 msgr2=0x7f1d4013dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:04.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.819+0000 7f1d45b89640 1 -- 172.21.15.100:0/3995125134 shutdown_connections 2024-09-13T11:57:04.858 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.819+0000 7f1d45b89640 1 -- 172.21.15.100:0/3995125134 wait complete. 2024-09-13T11:57:04.859 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T11:57:04.859 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-13T11:57:04.859 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-13T11:57:04.859 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-13T11:57:04.859 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b store 2024-09-13T11:57:04.859 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b sync_force --yes-i-really-mean-it 2024-09-13T11:57:04.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.956+0000 7fdb7e956640 10 monclient: get_monmap_and_config 2024-09-13T11:57:04.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.956+0000 7fdb7e956640 10 monclient: build_initial_monmap 2024-09-13T11:57:04.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.956+0000 7fdb7e956640 10 monclient: monmap: 2024-09-13T11:57:04.958 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:04.958 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.958 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:57:04.957719+0000 2024-09-13T11:57:04.959 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:57:04.957719+0000 2024-09-13T11:57:04.959 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:04.959 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.959 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:57:04.959 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:57:04.959 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:57:04.959 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:57:04.959 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:57:04.959 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.957+0000 7fdb7e956640 1 Processor -- start 2024-09-13T11:57:04.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.957+0000 7fdb7e956640 1 -- start start 2024-09-13T11:57:04.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.957+0000 7fdb7e956640 10 monclient: init 2024-09-13T11:57:04.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb7e956640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:04.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb7e956640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-13T11:57:04.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb7e956640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fdb780a3650 0x7fdb780a3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb7e956640 10 monclient(hunting): picked mon.noname-a con 0x7fdb780a3650 addr v2:172.21.15.100:3300/0 2024-09-13T11:57:04.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb7e956640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdb780a3f70 addr v1:172.21.15.122:6789/0 2024-09-13T11:57:04.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb7e956640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdb78135b10 addr v1:172.21.15.100:6789/0 2024-09-13T11:57:04.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb7e956640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb780a0d20 con 0x7fdb78135b10 2024-09-13T11:57:04.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb7e956640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb780a1000 con 0x7fdb780a3f70 2024-09-13T11:57:04.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb7e956640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb7e956640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7fdb78070f40 con 0x7fdb780a3650 2024-09-13T11:57:04.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb7e956640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:04.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb7e956640 10 monclient(hunting): authenticate will time out at 1005.551280s 2024-09-13T11:57:04.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb77fff640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fdb780a3650 0x7fdb780a3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb7cef3640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fdb78135b10 0x7fdb7813d870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:47494/0 (socket says 172.21.15.100:47494) 2024-09-13T11:57:04.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb7cef3640 1 -- 172.21.15.100:0/115604410 learned_addr learned my addr 172.21.15.100:0/115604410 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:57:04.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb77fff640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fdb780a3650 0x7fdb780a3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55488/0 (socket says 172.21.15.100:55488) 2024-09-13T11:57:04.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb77fff640 10 monclient(hunting): get_auth_request con 0x7fdb780a3650 auth_method 0 2024-09-13T11:57:04.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb77fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb77fff640 10 monclient(hunting): handle_auth_done global_id 14922 payload 274 2024-09-13T11:57:04.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb77fff640 1 -- 172.21.15.100:0/115604410 >> v1:172.21.15.122:6789/0 conn(0x7fdb780a3f70 legacy=0x7fdb78135400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:57:04.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb77fff640 1 -- 172.21.15.100:0/115604410 >> v1:172.21.15.100:6789/0 conn(0x7fdb78135b10 legacy=0x7fdb7813d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb77fff640 10 monclient: _finish_hunting 0 2024-09-13T11:57:04.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb77fff640 1 monclient: found mon.noname-a 2024-09-13T11:57:04.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:04.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb77fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:57:04.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb77fff640 1 -- 172.21.15.100:0/115604410 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb7809f9d0 con 0x7fdb780a3650 2024-09-13T11:57:04.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb76ffd640 1 -- 172.21.15.100:0/115604410 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1928698032 0 0) 0x7fdb780a0d20 con 0x7fdb78135b10 2024-09-13T11:57:04.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.958+0000 7fdb76ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:57:04.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.959+0000 7fdb77fff640 1 --2- 172.21.15.100:0/115604410 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb780a3650 0x7fdb780a3a30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdb680097a0 tx=0x7fdb6802a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c0e7961244c6987 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:04.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.959+0000 7fdb76ffd640 1 -- 172.21.15.100:0/115604410 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68037070 con 0x7fdb780a3650 2024-09-13T11:57:04.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.959+0000 7fdb76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.959+0000 7fdb76ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:57:04.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.959+0000 7fdb76ffd640 10 monclient: dump: 2024-09-13T11:57:04.971 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.971 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.971 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.972 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.972 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.972 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.972 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.972 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.972 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.972 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.959+0000 7fdb76ffd640 10 monclient: _finish_auth 0 2024-09-13T11:57:04.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.959+0000 7fdb76ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:57:04.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.959+0000 7fdb76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:04.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.959+0000 7fdb7e956640 5 monclient: authenticate success, global_id 14922 2024-09-13T11:57:04.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.959+0000 7fdb7e956640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:57:04.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.959+0000 7fdb76ffd640 1 -- 172.21.15.100:0/115604410 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb6802adb0 con 0x7fdb780a3650 2024-09-13T11:57:04.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.959+0000 7fdb76ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:04.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.959+0000 7fdb76ffd640 1 -- 172.21.15.100:0/115604410 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68032660 con 0x7fdb780a3650 2024-09-13T11:57:04.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.959+0000 7fdb7e956640 10 monclient: get_monmap_and_config success 2024-09-13T11:57:04.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.959+0000 7fdb7e956640 4 set_mon_vals no callback set 2024-09-13T11:57:04.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.959+0000 7fdb76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.959+0000 7fdb76ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:04.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.959+0000 7fdb76ffd640 10 monclient: dump: 2024-09-13T11:57:04.973 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.973 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.973 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.973 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.973 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.973 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.973 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.973 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.974 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.974 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.959+0000 7fdb7e956640 10 monclient: shutdown 2024-09-13T11:57:04.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.959+0000 7fdb7e956640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:04.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.959+0000 7fdb7e956640 1 -- 172.21.15.100:0/115604410 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb780a3650 msgr2=0x7fdb780a3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.959+0000 7fdb7e956640 1 --2- 172.21.15.100:0/115604410 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb780a3650 0x7fdb780a3a30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdb680097a0 tx=0x7fdb6802a060 comp rx=0 tx=0).stop 2024-09-13T11:57:04.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.960+0000 7fdb7e956640 1 -- 172.21.15.100:0/115604410 shutdown_connections 2024-09-13T11:57:04.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.960+0000 7fdb7e956640 1 --2- 172.21.15.100:0/115604410 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb780a3650 0x7fdb780a3a30 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.960+0000 7fdb7e956640 1 -- 172.21.15.100:0/115604410 >> 172.21.15.100:0/115604410 conn(0x7fdb780a8270 msgr2=0x7fdb780a4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:04.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.960+0000 7fdb7e956640 1 -- 172.21.15.100:0/115604410 shutdown_connections 2024-09-13T11:57:04.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.960+0000 7fdb7e956640 1 -- 172.21.15.100:0/115604410 wait complete. 2024-09-13T11:57:04.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.960+0000 7fdb7e956640 10 monclient: build_initial_monmap 2024-09-13T11:57:04.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.960+0000 7fdb7e956640 10 monclient: monmap: 2024-09-13T11:57:04.975 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:04.975 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:57:04.975 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:57:04.975 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:57:04.975 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:04.975 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.975 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:57:04.975 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:57:04.975 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:57:04.975 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.975 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.960+0000 7fdb7e956640 1 Processor -- start 2024-09-13T11:57:04.975 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7e956640 1 -- start start 2024-09-13T11:57:04.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7e956640 10 monclient: init 2024-09-13T11:57:04.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7e956640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:04.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7e956640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:57:04.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7e956640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb780a3650 0x7fdb781cac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7e956640 10 monclient(hunting): picked mon.noname-b con 0x7fdb780a3650 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:57:04.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7e956640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb780a3f70 0x7fdb78122150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7e956640 10 monclient(hunting): picked mon.noname-a con 0x7fdb780a3f70 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:57:04.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7e956640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb78135b10 0x7fdb78122690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7e956640 10 monclient(hunting): picked mon.noname-c con 0x7fdb78135b10 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:04.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7e956640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7e956640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fdb780a1ff0 con 0x7fdb780a3f70 2024-09-13T11:57:04.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7e956640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7e956640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fdb780a4380 con 0x7fdb780a3650 2024-09-13T11:57:04.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7e956640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7e956640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fdb781245a0 con 0x7fdb78135b10 2024-09-13T11:57:04.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7e956640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:04.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7e956640 10 monclient(hunting): authenticate will time out at 1005.554653s 2024-09-13T11:57:04.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb777fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb780a3f70 0x7fdb78122150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb777fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb780a3f70 0x7fdb78122150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55494/0 (socket says 172.21.15.100:55494) 2024-09-13T11:57:04.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb777fe640 1 -- 172.21.15.100:0/3623213175 learned_addr learned my addr 172.21.15.100:0/3623213175 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:04.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb777fe640 10 monclient(hunting): get_auth_request con 0x7fdb780a3f70 auth_method 0 2024-09-13T11:57:04.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7cef3640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb78135b10 0x7fdb78122690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb777fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7cef3640 10 monclient(hunting): get_auth_request con 0x7fdb78135b10 auth_method 0 2024-09-13T11:57:04.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7cef3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7cef3640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7cef3640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7cef3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7cef3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb7cef3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb77fff640 1 --2- 172.21.15.100:0/3623213175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb780a3650 0x7fdb781cac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb77fff640 10 monclient(hunting): get_auth_request con 0x7fdb780a3650 auth_method 0 2024-09-13T11:57:04.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb77fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.961+0000 7fdb777fe640 10 monclient(hunting): handle_auth_done global_id 14928 payload 824 2024-09-13T11:57:04.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb777fe640 1 -- 172.21.15.100:0/3623213175 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb78135b10 msgr2=0x7fdb78122690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb777fe640 1 --2- 172.21.15.100:0/3623213175 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb78135b10 0x7fdb78122690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb777fe640 1 -- 172.21.15.100:0/3623213175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb780a3650 msgr2=0x7fdb781cac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb777fe640 1 --2- 172.21.15.100:0/3623213175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb780a3650 0x7fdb781cac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb777fe640 10 monclient: _finish_hunting 0 2024-09-13T11:57:04.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb777fe640 1 monclient: found mon.noname-a 2024-09-13T11:57:04.980 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:04.980 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb777fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:57:04.980 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb777fe640 1 -- 172.21.15.100:0/3623213175 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb78076a90 con 0x7fdb780a3f70 2024-09-13T11:57:04.980 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb7cef3640 1 --2- 172.21.15.100:0/3623213175 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb78135b10 0x7fdb78122690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:57:04.980 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb77fff640 1 --2- 172.21.15.100:0/3623213175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb780a3650 0x7fdb781cac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:57:04.980 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb777fe640 1 --2- 172.21.15.100:0/3623213175 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb780a3f70 0x7fdb78122150 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c00c430 tx=0x7fdb6c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:04.980 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb74ff9640 1 -- 172.21.15.100:0/3623213175 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb6c011790 con 0x7fdb780a3f70 2024-09-13T11:57:04.980 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb74ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.980 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb74ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:57:04.980 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb74ff9640 10 monclient: dump: 2024-09-13T11:57:04.980 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.980 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.980 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.981 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.981 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.981 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.981 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.981 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.981 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.981 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb74ff9640 10 monclient: _finish_auth 0 2024-09-13T11:57:04.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb74ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:57:04.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb74ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:04.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb74ff9640 1 -- 172.21.15.100:0/3623213175 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb6c017070 con 0x7fdb780a3f70 2024-09-13T11:57:04.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb7e956640 5 monclient: authenticate success, global_id 14928 2024-09-13T11:57:04.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb7e956640 10 monclient: _renew_subs 2024-09-13T11:57:04.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb7e956640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:04.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb7e956640 1 -- 172.21.15.100:0/3623213175 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb78125c90 con 0x7fdb780a3f70 2024-09-13T11:57:04.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb74ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:04.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb74ff9640 1 -- 172.21.15.100:0/3623213175 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb6c011bb0 con 0x7fdb780a3f70 2024-09-13T11:57:04.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb74ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb74ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:04.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb74ff9640 10 monclient: dump: 2024-09-13T11:57:04.982 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.982 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.982 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.982 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.983 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.983 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.983 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.983 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.983 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.983 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb7e956640 10 monclient: _renew_subs 2024-09-13T11:57:04.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb767fc640 4 set_mon_vals no callback set 2024-09-13T11:57:04.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb7e956640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:04.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.962+0000 7fdb7e956640 1 -- 172.21.15.100:0/3623213175 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb781259b0 con 0x7fdb780a3f70 2024-09-13T11:57:04.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.963+0000 7fdb7e956640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:04.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.963+0000 7fdb74ff9640 1 -- 172.21.15.100:0/3623213175 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fdb6c01c460 con 0x7fdb780a3f70 2024-09-13T11:57:04.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.963+0000 7fdb7e956640 1 --2- 172.21.15.100:0/3623213175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb781cb1a0 0x7fdb78122bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.963+0000 7fdb7e956640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.963+0000 7fdb7e956640 1 -- 172.21.15.100:0/3623213175 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fdb780a2170 con 0x7fdb781cb1a0 2024-09-13T11:57:04.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.963+0000 7fdb74ff9640 1 --2- 172.21.15.100:0/3623213175 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fdb6403dfa0 0x7fdb64040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.963+0000 7fdb74ff9640 1 -- 172.21.15.100:0/3623213175 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(223..223 src has 1..223) ==== 4346+0+0 (secure 0 0 0) 0x7fdb6c052390 con 0x7fdb780a3f70 2024-09-13T11:57:04.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.963+0000 7fdb77fff640 1 --2- 172.21.15.100:0/3623213175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb781cb1a0 0x7fdb78122bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.963+0000 7fdb77fff640 10 monclient: get_auth_request con 0x7fdb781cb1a0 auth_method 0 2024-09-13T11:57:04.984 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.963+0000 7fdb77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.963+0000 7fdb77fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.963+0000 7fdb77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.963+0000 7fdb77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.963+0000 7fdb77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.963+0000 7fdb7cef3640 1 --2- 172.21.15.100:0/3623213175 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fdb6403dfa0 0x7fdb64040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.963+0000 7fdb77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.963+0000 7fdb7cef3640 10 monclient: get_auth_request con 0x7fdb6403dfa0 auth_method 0 2024-09-13T11:57:04.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.963+0000 7fdb77fff640 10 monclient(hunting): handle_auth_done global_id 5692 payload 824 2024-09-13T11:57:04.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.963+0000 7fdb77fff640 1 -- 172.21.15.100:0/3623213175 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdb780a5150 con 0x7fdb781cb1a0 2024-09-13T11:57:04.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.964+0000 7fdb77fff640 1 --2- 172.21.15.100:0/3623213175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb781cb1a0 0x7fdb78122bd0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fdb7809f420 tx=0x7fdb680046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:04.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.964+0000 7fdb7cef3640 1 --2- 172.21.15.100:0/3623213175 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fdb6403dfa0 0x7fdb64040460 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fdb70007dc0 tx=0x7fdb70007d10 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:04.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.964+0000 7fdb74ff9640 1 -- 172.21.15.100:0/3623213175 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68037040 con 0x7fdb781cb1a0 2024-09-13T11:57:04.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.964+0000 7fdb74ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.964+0000 7fdb74ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:57:04.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.964+0000 7fdb74ff9640 10 monclient: dump: 2024-09-13T11:57:04.986 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.986 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.986 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.986 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.986 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.986 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.986 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.987 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.987 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.987 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.964+0000 7fdb74ff9640 1 -- 172.21.15.100:0/3623213175 <== mon.1 v2:172.21.15.122:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdb68032e10 con 0x7fdb781cb1a0 2024-09-13T11:57:04.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.964+0000 7fdb74ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:04.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.964+0000 7fdb74ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:57:04.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.964+0000 7fdb74ff9640 1 -- 172.21.15.100:0/3623213175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb781cb1a0 msgr2=0x7fdb78122bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.964+0000 7fdb74ff9640 1 --2- 172.21.15.100:0/3623213175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb781cb1a0 0x7fdb78122bd0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fdb7809f420 tx=0x7fdb680046c0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.964+0000 7fdb74ff9640 1 -- 172.21.15.100:0/3623213175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb781cb1a0 msgr2=0x7fdb78122bd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:57:04.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.964+0000 7fdb74ff9640 1 --2- 172.21.15.100:0/3623213175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb781cb1a0 0x7fdb78122bd0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.971+0000 7fdb7e956640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-13T11:57:04.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.971+0000 7fdb7e956640 1 --2- 172.21.15.100:0/3623213175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb78123110 0x7fdb7811e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:04.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.971+0000 7fdb7e956640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:04.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.971+0000 7fdb7e956640 1 -- 172.21.15.100:0/3623213175 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7809e5a0 con 0x7fdb78123110 2024-09-13T11:57:04.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.971+0000 7fdb77fff640 1 --2- 172.21.15.100:0/3623213175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb78123110 0x7fdb7811e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:04.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.971+0000 7fdb77fff640 10 monclient: get_auth_request con 0x7fdb78123110 auth_method 0 2024-09-13T11:57:04.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.971+0000 7fdb77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:04.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.971+0000 7fdb77fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:04.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.971+0000 7fdb77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:04.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.971+0000 7fdb77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:04.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.971+0000 7fdb77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:04.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.971+0000 7fdb77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:04.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.972+0000 7fdb77fff640 10 monclient(hunting): handle_auth_done global_id 5695 payload 824 2024-09-13T11:57:04.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.972+0000 7fdb77fff640 1 -- 172.21.15.100:0/3623213175 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7fdb781234f0 con 0x7fdb78123110 2024-09-13T11:57:04.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.972+0000 7fdb77fff640 1 --2- 172.21.15.100:0/3623213175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb78123110 0x7fdb7811e290 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fdb680097a0 tx=0x7fdb68008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:04.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.972+0000 7fdb74ff9640 1 -- 172.21.15.100:0/3623213175 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68037040 con 0x7fdb78123110 2024-09-13T11:57:04.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.972+0000 7fdb74ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:04.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.972+0000 7fdb74ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:57:04.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.972+0000 7fdb74ff9640 10 monclient: dump: 2024-09-13T11:57:04.992 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:04.992 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:04.992 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.992 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:04.992 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:04.992 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:04.992 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:04.992 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:04.992 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:04.992 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:04.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.973+0000 7fdb74ff9640 1 -- 172.21.15.100:0/3623213175 <== mon.1 v2:172.21.15.122:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7fdb6803c940 con 0x7fdb78123110 2024-09-13T11:57:04.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.973+0000 7fdb74ff9640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-13T11:57:04.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.973+0000 7fdb74ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:57:04.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.973+0000 7fdb74ff9640 1 -- 172.21.15.100:0/3623213175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb78123110 msgr2=0x7fdb7811e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.973+0000 7fdb74ff9640 1 --2- 172.21.15.100:0/3623213175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb78123110 0x7fdb7811e290 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fdb680097a0 tx=0x7fdb68008030 comp rx=0 tx=0).stop 2024-09-13T11:57:04.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.973+0000 7fdb74ff9640 1 -- 172.21.15.100:0/3623213175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb78123110 msgr2=0x7fdb7811e290 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:57:04.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.973+0000 7fdb74ff9640 1 --2- 172.21.15.100:0/3623213175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb78123110 0x7fdb7811e290 secure :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fdb680097a0 tx=0x7fdb68008030 comp rx=0 tx=0).stop 2024-09-13T11:57:04.993 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:57:04.993 INFO:teuthology.orchestra.run.smithi100.stdout: "ret": 0, 2024-09-13T11:57:04.993 INFO:teuthology.orchestra.run.smithi100.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-13T11:57:04.993 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:57:04.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.973+0000 7fdb7e956640 1 -- 172.21.15.100:0/3623213175 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fdb6403dfa0 msgr2=0x7fdb64040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.973+0000 7fdb7e956640 1 --2- 172.21.15.100:0/3623213175 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fdb6403dfa0 0x7fdb64040460 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fdb70007dc0 tx=0x7fdb70007d10 comp rx=0 tx=0).stop 2024-09-13T11:57:04.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.973+0000 7fdb7e956640 10 monclient: shutdown 2024-09-13T11:57:04.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.973+0000 7fdb7e956640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:04.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.973+0000 7fdb7e956640 1 -- 172.21.15.100:0/3623213175 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb780a3f70 msgr2=0x7fdb78122150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:04.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.973+0000 7fdb7e956640 1 --2- 172.21.15.100:0/3623213175 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb780a3f70 0x7fdb78122150 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c00c430 tx=0x7fdb6c00c8f0 comp rx=0 tx=0).stop 2024-09-13T11:57:04.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.973+0000 7fdb7cef3640 1 -- 172.21.15.100:0/3623213175 reap_dead start 2024-09-13T11:57:04.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.973+0000 7fdb7cef3640 1 -- 172.21.15.100:0/3623213175 reap_dead start 2024-09-13T11:57:04.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.973+0000 7fdb7e956640 1 -- 172.21.15.100:0/3623213175 shutdown_connections 2024-09-13T11:57:04.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.973+0000 7fdb7e956640 1 -- 172.21.15.100:0/3623213175 >> 172.21.15.100:0/3623213175 conn(0x7fdb780a8270 msgr2=0x7fdb780a6360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:04.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.973+0000 7fdb7e956640 1 -- 172.21.15.100:0/3623213175 shutdown_connections 2024-09-13T11:57:04.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:04.973+0000 7fdb7e956640 1 -- 172.21.15.100:0/3623213175 wait complete. 2024-09-13T11:57:04.996 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-13T11:57:04.997 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-13T11:57:04.997 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:57:05.013 INFO:tasks.ceph.mon.b:Stopped 2024-09-13T11:57:05.013 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-13T11:57:05.013 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:57:05.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.107+0000 7f8525a31640 10 monclient: get_monmap_and_config 2024-09-13T11:57:05.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.107+0000 7f8525a31640 10 monclient: build_initial_monmap 2024-09-13T11:57:05.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.107+0000 7f8525a31640 10 monclient: monmap: 2024-09-13T11:57:05.109 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:05.109 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:05.109 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:57:05.108287+0000 2024-09-13T11:57:05.109 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:57:05.108287+0000 2024-09-13T11:57:05.109 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:05.109 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:05.109 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:57:05.109 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:57:05.109 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:57:05.109 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:57:05.109 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:57:05.109 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:05.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.107+0000 7f8525a31640 1 Processor -- start 2024-09-13T11:57:05.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f8525a31640 1 -- start start 2024-09-13T11:57:05.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f8525a31640 10 monclient: init 2024-09-13T11:57:05.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f8525a31640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:05.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f8525a31640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-13T11:57:05.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f8525a31640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8520137a60 addr v1:172.21.15.122:6789/0 2024-09-13T11:57:05.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f8525a31640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f85200a2980 0x7f852013c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:05.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f8525a31640 10 monclient(hunting): picked mon.noname-a con 0x7f85200a2980 addr v2:172.21.15.100:3300/0 2024-09-13T11:57:05.119 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f8525a31640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f852013ce90 0x7f852013f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:05.119 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f8525a31640 10 monclient(hunting): picked mon.noname-b con 0x7f852013ce90 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:05.119 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f8525a31640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8520138550 con 0x7f8520137a60 2024-09-13T11:57:05.119 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f8525a31640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:05.119 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f8525a31640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f85200a4410 con 0x7f85200a2980 2024-09-13T11:57:05.119 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f8525a31640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:05.119 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f8525a31640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f85200a5be0 con 0x7f852013ce90 2024-09-13T11:57:05.119 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f8525a31640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:05.119 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f8525a31640 10 monclient(hunting): authenticate will time out at 1005.701547s 2024-09-13T11:57:05.119 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851effd640 1 -- >> v1:172.21.15.122:6789/0 conn(0x7f8520137a60 legacy=0x7f8520137e40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6789/0 2024-09-13T11:57:05.119 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851f7fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f852013ce90 0x7f852013f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:05.120 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851e7fc640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f85200a2980 0x7f852013c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:05.120 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851f7fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f852013ce90 0x7f852013f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:46206/0 (socket says 172.21.15.100:46206) 2024-09-13T11:57:05.120 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851f7fe640 1 -- 172.21.15.100:0/4154483611 learned_addr learned my addr 172.21.15.100:0/4154483611 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:05.120 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851f7fe640 10 monclient(hunting): get_auth_request con 0x7f852013ce90 auth_method 0 2024-09-13T11:57:05.120 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:05.120 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:05.120 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:05.120 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851e7fc640 10 monclient(hunting): get_auth_request con 0x7f85200a2980 auth_method 0 2024-09-13T11:57:05.120 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:05.120 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:05.120 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:05.121 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:05.121 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:05.121 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:05.121 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:05.121 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:05.121 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:05.121 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851f7fe640 10 monclient(hunting): handle_auth_done global_id 24161 payload 274 2024-09-13T11:57:05.121 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851f7fe640 1 -- 172.21.15.100:0/4154483611 >> v2:172.21.15.100:3300/0 conn(0x7f85200a2980 msgr2=0x7f852013c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:57:05.121 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851f7fe640 1 --2- 172.21.15.100:0/4154483611 >> v2:172.21.15.100:3300/0 conn(0x7f85200a2980 0x7f852013c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:05.121 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851f7fe640 1 -- 172.21.15.100:0/4154483611 >> v1:172.21.15.122:6789/0 conn(0x7f8520137a60 legacy=0x7f8520137e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:57:05.122 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851f7fe640 10 monclient: _finish_hunting 0 2024-09-13T11:57:05.122 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851f7fe640 1 monclient: found mon.noname-b 2024-09-13T11:57:05.122 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:05.122 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.108+0000 7f851f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:57:05.122 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.109+0000 7f851f7fe640 1 -- 172.21.15.100:0/4154483611 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85200a76b0 con 0x7f852013ce90 2024-09-13T11:57:05.122 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.109+0000 7f851e7fc640 1 --2- 172.21.15.100:0/4154483611 >> v2:172.21.15.100:3300/0 conn(0x7f85200a2980 0x7f852013c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:57:05.122 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.109+0000 7f851f7fe640 1 --2- 172.21.15.100:0/4154483611 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f852013ce90 0x7f852013f280 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8518009e50 tx=0x7f851802a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e414548155222c60 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:05.122 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.109+0000 7f851dffb640 1 -- 172.21.15.100:0/4154483611 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8518036070 con 0x7f852013ce90 2024-09-13T11:57:05.122 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.109+0000 7f851dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:05.122 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.109+0000 7f851dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:57:05.122 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.109+0000 7f851dffb640 10 monclient: dump: 2024-09-13T11:57:05.123 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:05.123 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:05.123 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:05.123 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:05.123 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:05.123 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:05.123 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:05.123 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:05.123 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:05.123 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:05.124 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.109+0000 7f851dffb640 10 monclient: _finish_auth 0 2024-09-13T11:57:05.124 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.109+0000 7f851dffb640 10 monclient: _check_auth_tickets 2024-09-13T11:57:05.124 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.109+0000 7f851dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:05.124 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.109+0000 7f851dffb640 1 -- 172.21.15.100:0/4154483611 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8518031600 con 0x7f852013ce90 2024-09-13T11:57:05.124 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.109+0000 7f851dffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:05.124 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.109+0000 7f851dffb640 1 -- 172.21.15.100:0/4154483611 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85180319d0 con 0x7f852013ce90 2024-09-13T11:57:05.124 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.109+0000 7f8525a31640 5 monclient: authenticate success, global_id 24161 2024-09-13T11:57:05.125 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.109+0000 7f8525a31640 10 monclient: get_monmap_and_config success 2024-09-13T11:57:05.125 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.109+0000 7f8525a31640 4 set_mon_vals no callback set 2024-09-13T11:57:05.125 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.109+0000 7f851dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:05.125 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.109+0000 7f851dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:57:05.125 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.109+0000 7f851dffb640 10 monclient: dump: 2024-09-13T11:57:05.125 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:05.125 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:05.125 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:05.126 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:05.126 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:05.126 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:05.126 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:05.126 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:05.126 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:05.126 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:05.126 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.110+0000 7f8525a31640 10 monclient: shutdown 2024-09-13T11:57:05.126 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.110+0000 7f8525a31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:05.126 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.110+0000 7f8525a31640 1 -- 172.21.15.100:0/4154483611 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f852013ce90 msgr2=0x7f852013f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:05.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.110+0000 7f8525a31640 1 --2- 172.21.15.100:0/4154483611 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f852013ce90 0x7f852013f280 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8518009e50 tx=0x7f851802a560 comp rx=0 tx=0).stop 2024-09-13T11:57:05.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.110+0000 7f8525a31640 1 -- 172.21.15.100:0/4154483611 shutdown_connections 2024-09-13T11:57:05.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.110+0000 7f8525a31640 1 --2- 172.21.15.100:0/4154483611 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f852013ce90 0x7f852013f280 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:05.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.110+0000 7f8525a31640 1 --2- 172.21.15.100:0/4154483611 >> v2:172.21.15.100:3300/0 conn(0x7f85200a2980 0x7f852013c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:05.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.110+0000 7f8525a31640 1 -- 172.21.15.100:0/4154483611 >> 172.21.15.100:0/4154483611 conn(0x7f85200aa580 msgr2=0x7f85200a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:05.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.110+0000 7f8525a31640 1 -- 172.21.15.100:0/4154483611 shutdown_connections 2024-09-13T11:57:05.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.110+0000 7f8525a31640 1 -- 172.21.15.100:0/4154483611 wait complete. 2024-09-13T11:57:05.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.111+0000 7f8525a31640 10 monclient: build_initial_monmap 2024-09-13T11:57:05.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.111+0000 7f8525a31640 10 monclient: monmap: 2024-09-13T11:57:05.128 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:05.128 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:57:05.128 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:57:05.128 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:57:05.128 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:05.128 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:05.128 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:57:05.128 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:57:05.128 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:57:05.128 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:05.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.111+0000 7f8525a31640 1 Processor -- start 2024-09-13T11:57:05.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.111+0000 7f8525a31640 1 -- start start 2024-09-13T11:57:05.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.111+0000 7f8525a31640 10 monclient: init 2024-09-13T11:57:05.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.111+0000 7f8525a31640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:05.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.111+0000 7f8525a31640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:57:05.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.111+0000 7f8525a31640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85200a2980 0x7f8520136100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:05.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.111+0000 7f8525a31640 10 monclient(hunting): picked mon.noname-a con 0x7f85200a2980 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:57:05.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.111+0000 7f8525a31640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8520137a60 0x7f8520136640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:05.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.111+0000 7f8525a31640 10 monclient(hunting): picked mon.noname-b con 0x7f8520137a60 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:57:05.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8525a31640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f852013ce90 0x7f852009d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:05.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8525a31640 10 monclient(hunting): picked mon.noname-c con 0x7f852013ce90 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:05.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8525a31640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:05.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8525a31640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f852006fd80 con 0x7f85200a2980 2024-09-13T11:57:05.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8525a31640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:05.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8525a31640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f85200a1700 con 0x7f8520137a60 2024-09-13T11:57:05.129 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8525a31640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:05.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8525a31640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f852006ff00 con 0x7f852013ce90 2024-09-13T11:57:05.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8525a31640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:05.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8525a31640 10 monclient(hunting): authenticate will time out at 1005.705037s 2024-09-13T11:57:05.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851effd640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85200a2980 0x7f8520136100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:05.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851effd640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85200a2980 0x7f8520136100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55520/0 (socket says 172.21.15.100:55520) 2024-09-13T11:57:05.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851effd640 1 -- 172.21.15.100:0/331577564 learned_addr learned my addr 172.21.15.100:0/331577564 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:05.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851effd640 10 monclient(hunting): get_auth_request con 0x7f85200a2980 auth_method 0 2024-09-13T11:57:05.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:05.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:05.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:05.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851f7fe640 1 --2- 172.21.15.100:0/331577564 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f852013ce90 0x7f852009d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:05.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851e7fc640 1 -- 172.21.15.100:0/331577564 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8520137a60 msgr2=0x7f8520136640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.122:3300/0 2024-09-13T11:57:05.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851f7fe640 10 monclient(hunting): get_auth_request con 0x7f852013ce90 auth_method 0 2024-09-13T11:57:05.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851e7fc640 1 --2- 172.21.15.100:0/331577564 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8520137a60 0x7f8520136640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:57:05.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:05.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:05.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:05.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:05.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:05.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:05.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:05.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:05.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:05.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851effd640 10 monclient(hunting): handle_auth_done global_id 14934 payload 824 2024-09-13T11:57:05.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851effd640 1 -- 172.21.15.100:0/331577564 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f852013ce90 msgr2=0x7f852009d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:05.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851effd640 1 --2- 172.21.15.100:0/331577564 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f852013ce90 0x7f852009d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:05.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851effd640 1 -- 172.21.15.100:0/331577564 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8520137a60 msgr2=0x7f8520136640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:57:05.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851effd640 1 --2- 172.21.15.100:0/331577564 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8520137a60 0x7f8520136640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:05.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851effd640 10 monclient: _finish_hunting 0 2024-09-13T11:57:05.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851effd640 1 monclient: found mon.noname-a 2024-09-13T11:57:05.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:05.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:57:05.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851effd640 1 -- 172.21.15.100:0/331577564 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8520120ad0 con 0x7f85200a2980 2024-09-13T11:57:05.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851f7fe640 1 --2- 172.21.15.100:0/331577564 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f852013ce90 0x7f852009d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:57:05.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851effd640 1 --2- 172.21.15.100:0/331577564 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85200a2980 0x7f8520136100 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f851000b550 tx=0x7f851000ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:05.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8507fff640 1 -- 172.21.15.100:0/331577564 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8510003d70 con 0x7f85200a2980 2024-09-13T11:57:05.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8507fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:05.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8507fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:57:05.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8507fff640 10 monclient: dump: 2024-09-13T11:57:05.133 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:05.133 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:05.133 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:05.133 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:05.133 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:05.133 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:05.133 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:05.133 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:05.133 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:05.133 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:05.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8507fff640 10 monclient: _finish_auth 0 2024-09-13T11:57:05.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8507fff640 10 monclient: _check_auth_tickets 2024-09-13T11:57:05.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8507fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:05.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8507fff640 1 -- 172.21.15.100:0/331577564 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8510015070 con 0x7f85200a2980 2024-09-13T11:57:05.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8525a31640 5 monclient: authenticate success, global_id 14934 2024-09-13T11:57:05.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8507fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:05.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8507fff640 1 -- 172.21.15.100:0/331577564 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85100052b0 con 0x7f85200a2980 2024-09-13T11:57:05.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8507fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:05.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8507fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:05.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8507fff640 10 monclient: dump: 2024-09-13T11:57:05.134 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:05.134 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:05.134 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:05.135 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:05.135 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:05.135 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:05.135 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:05.135 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:05.135 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:05.135 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:05.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8525a31640 10 monclient: _renew_subs 2024-09-13T11:57:05.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8525a31640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:05.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8525a31640 1 -- 172.21.15.100:0/331577564 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85200a8250 con 0x7f85200a2980 2024-09-13T11:57:05.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f851d7fa640 4 set_mon_vals no callback set 2024-09-13T11:57:05.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8525a31640 10 monclient: _renew_subs 2024-09-13T11:57:05.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8525a31640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:05.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.112+0000 7f8525a31640 1 -- 172.21.15.100:0/331577564 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85201207f0 con 0x7f85200a2980 2024-09-13T11:57:05.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.113+0000 7f8507fff640 1 -- 172.21.15.100:0/331577564 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f8510005450 con 0x7f85200a2980 2024-09-13T11:57:05.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.113+0000 7f8525a31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:05.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.113+0000 7f8525a31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:05.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.113+0000 7f8525a31640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:05.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.113+0000 7f8525a31640 1 -- 172.21.15.100:0/331577564 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8520137e40 con 0x7f85200a2980 2024-09-13T11:57:05.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.113+0000 7f8507fff640 1 --2- 172.21.15.100:0/331577564 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f850c03df50 0x7f850c040410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:05.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.113+0000 7f8507fff640 1 -- 172.21.15.100:0/331577564 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(223..223 src has 1..223) ==== 4346+0+0 (secure 0 0 0) 0x7f85100523e0 con 0x7f85200a2980 2024-09-13T11:57:05.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.114+0000 7f851e7fc640 1 --2- 172.21.15.100:0/331577564 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f850c03df50 0x7f850c040410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:05.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.114+0000 7f851e7fc640 10 monclient: get_auth_request con 0x7f850c03df50 auth_method 0 2024-09-13T11:57:05.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.114+0000 7f851e7fc640 1 --2- 172.21.15.100:0/331577564 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f850c03df50 0x7f850c040410 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8514009630 tx=0x7f8514009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:05.137 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.118+0000 7f8507fff640 1 -- 172.21.15.100:0/331577564 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f8510019ce0 con 0x7f85200a2980 2024-09-13T11:57:05.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.118+0000 7f8507fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:05.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.118+0000 7f8507fff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:57:05.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.357+0000 7f8525a31640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:57:05.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.357+0000 7f8525a31640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:57:05.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.357+0000 7f8525a31640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:05.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.357+0000 7f8525a31640 1 -- 172.21.15.100:0/331577564 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f852006a0f0 con 0x7f85200a2980 2024-09-13T11:57:05.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.358+0000 7f8507fff640 1 -- 172.21.15.100:0/331577564 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f85100520b0 con 0x7f85200a2980 2024-09-13T11:57:05.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.358+0000 7f8507fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:57:05.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.358+0000 7f8507fff640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:57:05.360 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:57:05.361 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.360+0000 7f8525a31640 1 -- 172.21.15.100:0/331577564 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f850c03df50 msgr2=0x7f850c040410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:05.361 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.360+0000 7f8525a31640 1 --2- 172.21.15.100:0/331577564 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f850c03df50 0x7f850c040410 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8514009630 tx=0x7f8514009190 comp rx=0 tx=0).stop 2024-09-13T11:57:05.361 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.360+0000 7f8525a31640 10 monclient: shutdown 2024-09-13T11:57:05.361 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.360+0000 7f8525a31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:05.361 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.360+0000 7f8525a31640 1 -- 172.21.15.100:0/331577564 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85200a2980 msgr2=0x7f8520136100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:05.362 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.360+0000 7f8525a31640 1 --2- 172.21.15.100:0/331577564 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85200a2980 0x7f8520136100 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f851000b550 tx=0x7f851000ba10 comp rx=0 tx=0).stop 2024-09-13T11:57:05.362 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.360+0000 7f8525a31640 1 -- 172.21.15.100:0/331577564 shutdown_connections 2024-09-13T11:57:05.362 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.360+0000 7f8525a31640 1 --2- 172.21.15.100:0/331577564 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f850c03df50 0x7f850c040410 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:05.362 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.360+0000 7f8525a31640 1 --2- 172.21.15.100:0/331577564 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f852013ce90 0x7f852009d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:05.362 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.360+0000 7f8525a31640 1 --2- 172.21.15.100:0/331577564 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8520137a60 0x7f8520136640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:05.362 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.360+0000 7f8525a31640 1 --2- 172.21.15.100:0/331577564 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85200a2980 0x7f8520136100 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:05.362 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.360+0000 7f8525a31640 1 -- 172.21.15.100:0/331577564 >> 172.21.15.100:0/331577564 conn(0x7f85200aa580 msgr2=0x7f85200a3620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:05.362 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.360+0000 7f8525a31640 1 -- 172.21.15.100:0/331577564 shutdown_connections 2024-09-13T11:57:05.362 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:05.360+0000 7f8525a31640 1 -- 172.21.15.100:0/331577564 wait complete. 2024-09-13T11:57:08.372 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:57:08.479 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.477+0000 7f75202f7640 10 monclient: get_monmap_and_config 2024-09-13T11:57:08.479 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.477+0000 7f75202f7640 10 monclient: build_initial_monmap 2024-09-13T11:57:08.480 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.477+0000 7f75202f7640 10 monclient: monmap: 2024-09-13T11:57:08.480 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:08.480 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:08.480 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:57:08.479420+0000 2024-09-13T11:57:08.480 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:57:08.479420+0000 2024-09-13T11:57:08.480 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:08.480 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:08.480 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:57:08.480 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:57:08.480 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:57:08.480 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:57:08.481 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:57:08.481 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:08.487 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.478+0000 7f75202f7640 1 Processor -- start 2024-09-13T11:57:08.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.478+0000 7f75202f7640 1 -- start start 2024-09-13T11:57:08.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.478+0000 7f75202f7640 10 monclient: init 2024-09-13T11:57:08.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.478+0000 7f75202f7640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:08.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.478+0000 7f75202f7640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-13T11:57:08.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.478+0000 7f75202f7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f75180a37d0 0x7f75180a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:08.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.478+0000 7f75202f7640 10 monclient(hunting): picked mon.noname-c con 0x7f75180a37d0 addr v2:172.21.15.122:3300/0 2024-09-13T11:57:08.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.478+0000 7f75202f7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f751806eeb0 addr v1:172.21.15.100:6789/0 2024-09-13T11:57:08.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.478+0000 7f75202f7640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f75180a40f0 0x7f7518138080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:08.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.478+0000 7f75202f7640 10 monclient(hunting): picked mon.noname-a con 0x7f75180a40f0 addr v2:172.21.15.100:3300/0 2024-09-13T11:57:08.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f75202f7640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75180a45a0 con 0x7f751806eeb0 2024-09-13T11:57:08.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f75202f7640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:08.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f75202f7640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f751809fde0 con 0x7f75180a40f0 2024-09-13T11:57:08.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f75202f7640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:08.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f75202f7640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f751806f330 con 0x7f75180a37d0 2024-09-13T11:57:08.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f75202f7640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:08.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f75202f7640 10 monclient(hunting): authenticate will time out at 1009.073042s 2024-09-13T11:57:08.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f751e093640 1 -- >> v2:172.21.15.122:3300/0 conn(0x7f75180a37d0 msgr2=0x7f75180a3bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.122:3300/0 2024-09-13T11:57:08.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f751e093640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f75180a37d0 0x7f75180a3bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:57:08.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f751e894640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f75180a40f0 0x7f7518138080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:08.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f751e894640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f75180a40f0 0x7f7518138080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55526/0 (socket says 172.21.15.100:55526) 2024-09-13T11:57:08.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f751e894640 1 -- 172.21.15.100:0/834377423 learned_addr learned my addr 172.21.15.100:0/834377423 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:08.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f751e894640 10 monclient(hunting): get_auth_request con 0x7f75180a40f0 auth_method 0 2024-09-13T11:57:08.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f751e894640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:08.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f751e894640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:08.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f751e894640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:08.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f751e894640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:08.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f751e894640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:08.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f751e894640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:08.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f751e894640 10 monclient(hunting): handle_auth_done global_id 14937 payload 274 2024-09-13T11:57:08.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f751e894640 1 -- 172.21.15.100:0/834377423 >> v2:172.21.15.122:3300/0 conn(0x7f75180a37d0 msgr2=0x7f75180a3bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:57:08.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f751e894640 1 --2- 172.21.15.100:0/834377423 >> v2:172.21.15.122:3300/0 conn(0x7f75180a37d0 0x7f75180a3bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:08.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f751e894640 1 -- 172.21.15.100:0/834377423 >> v1:172.21.15.100:6789/0 conn(0x7f751806eeb0 legacy=0x7f7518137970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:08.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f751e894640 10 monclient: _finish_hunting 0 2024-09-13T11:57:08.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f751e894640 1 monclient: found mon.noname-a 2024-09-13T11:57:08.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f751e894640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:08.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f751e894640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:57:08.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.479+0000 7f751e894640 1 -- 172.21.15.100:0/834377423 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f751806e950 con 0x7f75180a40f0 2024-09-13T11:57:08.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f751e894640 1 --2- 172.21.15.100:0/834377423 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75180a40f0 0x7f7518138080 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f75180a55a0 tx=0x7f751002a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28257e894a975b14 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:08.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f751d091640 1 -- 172.21.15.100:0/834377423 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7510036070 con 0x7f75180a40f0 2024-09-13T11:57:08.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f751d091640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:08.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f751d091640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:57:08.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f751d091640 10 monclient: dump: 2024-09-13T11:57:08.491 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:08.491 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:08.491 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:08.491 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:08.491 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:08.491 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:08.491 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:08.492 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:08.492 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:08.492 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:08.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f751d091640 10 monclient: _finish_auth 0 2024-09-13T11:57:08.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f751d091640 10 monclient: _check_auth_tickets 2024-09-13T11:57:08.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f751d091640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:08.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f751d091640 1 -- 172.21.15.100:0/834377423 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f751002ae50 con 0x7f75180a40f0 2024-09-13T11:57:08.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f75202f7640 5 monclient: authenticate success, global_id 14937 2024-09-13T11:57:08.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f75202f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:57:08.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f751d091640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:08.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f75202f7640 10 monclient: get_monmap_and_config success 2024-09-13T11:57:08.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f751d091640 1 -- 172.21.15.100:0/834377423 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7510031820 con 0x7f75180a40f0 2024-09-13T11:57:08.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f751d091640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:08.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f75202f7640 4 set_mon_vals no callback set 2024-09-13T11:57:08.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f751d091640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:08.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f751d091640 10 monclient: dump: 2024-09-13T11:57:08.493 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:08.493 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:08.493 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:08.493 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:08.493 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:08.493 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:08.493 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:08.493 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:08.493 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:08.493 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:08.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f75202f7640 10 monclient: shutdown 2024-09-13T11:57:08.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f75202f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:08.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f75202f7640 1 -- 172.21.15.100:0/834377423 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75180a40f0 msgr2=0x7f7518138080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:08.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f75202f7640 1 --2- 172.21.15.100:0/834377423 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75180a40f0 0x7f7518138080 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f75180a55a0 tx=0x7f751002a630 comp rx=0 tx=0).stop 2024-09-13T11:57:08.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f75202f7640 1 -- 172.21.15.100:0/834377423 shutdown_connections 2024-09-13T11:57:08.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f75202f7640 1 --2- 172.21.15.100:0/834377423 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75180a40f0 0x7f7518138080 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:08.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f75202f7640 1 --2- 172.21.15.100:0/834377423 >> v2:172.21.15.122:3300/0 conn(0x7f75180a37d0 0x7f75180a3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:08.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f75202f7640 1 -- 172.21.15.100:0/834377423 >> 172.21.15.100:0/834377423 conn(0x7f75180a83f0 msgr2=0x7f75180a48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:08.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f75202f7640 1 -- 172.21.15.100:0/834377423 shutdown_connections 2024-09-13T11:57:08.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.480+0000 7f75202f7640 1 -- 172.21.15.100:0/834377423 wait complete. 2024-09-13T11:57:08.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.481+0000 7f75202f7640 10 monclient: build_initial_monmap 2024-09-13T11:57:08.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.481+0000 7f75202f7640 10 monclient: monmap: 2024-09-13T11:57:08.494 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:08.495 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:57:08.495 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:57:08.495 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:57:08.495 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:08.495 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:08.495 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:57:08.495 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:57:08.495 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:57:08.495 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:08.495 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.481+0000 7f75202f7640 1 Processor -- start 2024-09-13T11:57:08.495 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.481+0000 7f75202f7640 1 -- start start 2024-09-13T11:57:08.495 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.481+0000 7f75202f7640 10 monclient: init 2024-09-13T11:57:08.495 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f75202f7640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:08.496 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f75202f7640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:57:08.496 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f75202f7640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f751806eeb0 0x7f75181ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:08.496 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f75202f7640 10 monclient(hunting): picked mon.noname-c con 0x7f751806eeb0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:08.496 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f75202f7640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75180a37d0 0x7f75181cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:08.496 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f75202f7640 10 monclient(hunting): picked mon.noname-a con 0x7f75180a37d0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:57:08.496 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f75202f7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f75180a40f0 0x7f75181241d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:08.496 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f75202f7640 10 monclient(hunting): picked mon.noname-b con 0x7f75180a40f0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:57:08.496 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f75202f7640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:08.496 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f75202f7640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f751809edd0 con 0x7f75180a37d0 2024-09-13T11:57:08.496 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f75202f7640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:08.496 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f75202f7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f751809ec50 con 0x7f75180a40f0 2024-09-13T11:57:08.496 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f75202f7640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:08.496 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f75202f7640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f75181385c0 con 0x7f751806eeb0 2024-09-13T11:57:08.497 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f75202f7640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:08.497 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f75202f7640 10 monclient(hunting): authenticate will time out at 1009.076458s 2024-09-13T11:57:08.497 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751e093640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f751806eeb0 0x7f75181ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:08.497 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751d892640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75180a37d0 0x7f75181cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:08.497 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751e093640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f751806eeb0 0x7f75181ccc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:46230/0 (socket says 172.21.15.100:46230) 2024-09-13T11:57:08.497 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751e093640 1 -- 172.21.15.100:0/1680558877 learned_addr learned my addr 172.21.15.100:0/1680558877 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:08.497 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751e093640 10 monclient(hunting): get_auth_request con 0x7f751806eeb0 auth_method 0 2024-09-13T11:57:08.497 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751e093640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:08.497 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751e093640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:08.497 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751d892640 1 --2- 172.21.15.100:0/1680558877 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75180a37d0 0x7f75181cd1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55540/0 (socket says 172.21.15.100:55540) 2024-09-13T11:57:08.497 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751e093640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:08.497 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751d892640 10 monclient(hunting): get_auth_request con 0x7f75180a37d0 auth_method 0 2024-09-13T11:57:08.497 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751d892640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:08.498 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751d892640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:08.498 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751d892640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:08.498 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751e894640 1 -- 172.21.15.100:0/1680558877 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f75180a40f0 msgr2=0x7f75181241d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.122:3300/0 2024-09-13T11:57:08.498 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751e894640 1 --2- 172.21.15.100:0/1680558877 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f75180a40f0 0x7f75181241d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:57:08.498 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751d892640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:08.498 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751d892640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:08.498 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751d892640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:08.498 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751e093640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:08.498 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751e093640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:08.498 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751e093640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:08.498 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751d892640 10 monclient(hunting): handle_auth_done global_id 14940 payload 824 2024-09-13T11:57:08.498 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751d892640 1 -- 172.21.15.100:0/1680558877 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f751806eeb0 msgr2=0x7f75181ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:08.498 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751d892640 1 --2- 172.21.15.100:0/1680558877 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f751806eeb0 0x7f75181ccc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:08.499 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751d892640 1 -- 172.21.15.100:0/1680558877 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f75180a40f0 msgr2=0x7f75181241d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:57:08.499 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751d892640 1 --2- 172.21.15.100:0/1680558877 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f75180a40f0 0x7f75181241d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:08.499 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751d892640 10 monclient: _finish_hunting 0 2024-09-13T11:57:08.499 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751d892640 1 monclient: found mon.noname-a 2024-09-13T11:57:08.499 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751d892640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:08.499 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751d892640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:57:08.499 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751d892640 1 -- 172.21.15.100:0/1680558877 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7518127ff0 con 0x7f75180a37d0 2024-09-13T11:57:08.499 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.482+0000 7f751d892640 1 --2- 172.21.15.100:0/1680558877 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75180a37d0 0x7f75181cd1b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f750800c3c0 tx=0x7f750800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:08.499 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.483+0000 7f750effd640 1 -- 172.21.15.100:0/1680558877 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7508017070 con 0x7f75180a37d0 2024-09-13T11:57:08.499 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.483+0000 7f750effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:08.499 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.483+0000 7f750effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:57:08.499 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.483+0000 7f750effd640 10 monclient: dump: 2024-09-13T11:57:08.500 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:08.500 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:08.500 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:08.500 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:08.500 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:08.500 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:08.500 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:08.500 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:08.500 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:08.500 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:08.500 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.483+0000 7f750effd640 10 monclient: _finish_auth 0 2024-09-13T11:57:08.500 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.483+0000 7f750effd640 10 monclient: _check_auth_tickets 2024-09-13T11:57:08.500 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.483+0000 7f750effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:08.501 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.483+0000 7f750effd640 1 -- 172.21.15.100:0/1680558877 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75080052d0 con 0x7f75180a37d0 2024-09-13T11:57:08.501 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.483+0000 7f75202f7640 5 monclient: authenticate success, global_id 14940 2024-09-13T11:57:08.501 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.483+0000 7f75202f7640 10 monclient: _renew_subs 2024-09-13T11:57:08.501 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.483+0000 7f75202f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:08.501 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.483+0000 7f75202f7640 1 -- 172.21.15.100:0/1680558877 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7518125c90 con 0x7f75180a37d0 2024-09-13T11:57:08.501 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.483+0000 7f750effd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:08.501 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.483+0000 7f750effd640 1 -- 172.21.15.100:0/1680558877 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7508003830 con 0x7f75180a37d0 2024-09-13T11:57:08.501 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.483+0000 7f750effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:08.501 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.483+0000 7f750effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:08.501 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.483+0000 7f750effd640 10 monclient: dump: 2024-09-13T11:57:08.501 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:08.501 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:08.501 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:08.501 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:08.502 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:08.502 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:08.502 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:08.502 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:08.502 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:08.502 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:08.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.483+0000 7f751c890640 4 set_mon_vals no callback set 2024-09-13T11:57:08.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.483+0000 7f75202f7640 10 monclient: _renew_subs 2024-09-13T11:57:08.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.483+0000 7f75202f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:08.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.483+0000 7f75202f7640 1 -- 172.21.15.100:0/1680558877 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7518127a30 con 0x7f75180a37d0 2024-09-13T11:57:08.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.484+0000 7f750effd640 1 -- 172.21.15.100:0/1680558877 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f750801a070 con 0x7f75180a37d0 2024-09-13T11:57:08.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.484+0000 7f75202f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:08.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.484+0000 7f75202f7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:08.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.484+0000 7f75202f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:08.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.484+0000 7f75202f7640 1 -- 172.21.15.100:0/1680558877 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7518137930 con 0x7f75180a37d0 2024-09-13T11:57:08.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.484+0000 7f750effd640 1 --2- 172.21.15.100:0/1680558877 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f74fc03df50 0x7f74fc040410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:08.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.484+0000 7f750effd640 1 -- 172.21.15.100:0/1680558877 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(223..223 src has 1..223) ==== 4346+0+0 (secure 0 0 0) 0x7f75080517e0 con 0x7f75180a37d0 2024-09-13T11:57:08.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.484+0000 7f751e093640 1 --2- 172.21.15.100:0/1680558877 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f74fc03df50 0x7f74fc040410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:08.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.484+0000 7f751e093640 10 monclient: get_auth_request con 0x7f74fc03df50 auth_method 0 2024-09-13T11:57:08.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.485+0000 7f751e093640 1 --2- 172.21.15.100:0/1680558877 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f74fc03df50 0x7f74fc040410 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7514009450 tx=0x7f75140093e0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:08.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.489+0000 7f750effd640 1 -- 172.21.15.100:0/1680558877 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f7508010db0 con 0x7f75180a37d0 2024-09-13T11:57:08.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.489+0000 7f750effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:08.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.489+0000 7f750effd640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:57:08.732 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.730+0000 7f75202f7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:57:08.732 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.730+0000 7f75202f7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:57:08.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.730+0000 7f75202f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:08.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.730+0000 7f75202f7640 1 -- 172.21.15.100:0/1680558877 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f75180a3bb0 con 0x7f75180a37d0 2024-09-13T11:57:08.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.730+0000 7f750effd640 1 -- 172.21.15.100:0/1680558877 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f7508019dc0 con 0x7f75180a37d0 2024-09-13T11:57:08.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.730+0000 7f750effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:57:08.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.730+0000 7f750effd640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:57:08.733 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:57:08.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.732+0000 7f75202f7640 1 -- 172.21.15.100:0/1680558877 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f74fc03df50 msgr2=0x7f74fc040410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:08.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.732+0000 7f75202f7640 1 --2- 172.21.15.100:0/1680558877 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f74fc03df50 0x7f74fc040410 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7514009450 tx=0x7f75140093e0 comp rx=0 tx=0).stop 2024-09-13T11:57:08.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.732+0000 7f75202f7640 10 monclient: shutdown 2024-09-13T11:57:08.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.732+0000 7f75202f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:08.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.732+0000 7f75202f7640 1 -- 172.21.15.100:0/1680558877 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75180a37d0 msgr2=0x7f75181cd1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:08.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.732+0000 7f75202f7640 1 --2- 172.21.15.100:0/1680558877 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75180a37d0 0x7f75181cd1b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f750800c3c0 tx=0x7f750800c880 comp rx=0 tx=0).stop 2024-09-13T11:57:08.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.732+0000 7f75202f7640 1 -- 172.21.15.100:0/1680558877 shutdown_connections 2024-09-13T11:57:08.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.732+0000 7f75202f7640 1 --2- 172.21.15.100:0/1680558877 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f74fc03df50 0x7f74fc040410 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:08.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.732+0000 7f75202f7640 1 --2- 172.21.15.100:0/1680558877 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75180a37d0 0x7f75181cd1b0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:08.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.732+0000 7f75202f7640 1 --2- 172.21.15.100:0/1680558877 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f75180a40f0 0x7f75181241d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:08.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.732+0000 7f75202f7640 1 --2- 172.21.15.100:0/1680558877 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f751806eeb0 0x7f75181ccc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:08.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.732+0000 7f75202f7640 1 -- 172.21.15.100:0/1680558877 >> 172.21.15.100:0/1680558877 conn(0x7f75180a83f0 msgr2=0x7f7518137d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:08.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.732+0000 7f75202f7640 1 -- 172.21.15.100:0/1680558877 shutdown_connections 2024-09-13T11:57:08.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:08.732+0000 7f75202f7640 1 -- 172.21.15.100:0/1680558877 wait complete. 2024-09-13T11:57:11.746 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:57:11.853 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.852+0000 7fd325394640 10 monclient: get_monmap_and_config 2024-09-13T11:57:11.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.852+0000 7fd325394640 10 monclient: build_initial_monmap 2024-09-13T11:57:11.854 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.852+0000 7fd325394640 10 monclient: monmap: 2024-09-13T11:57:11.854 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:11.854 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:11.854 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:57:11.853501+0000 2024-09-13T11:57:11.854 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:57:11.853501+0000 2024-09-13T11:57:11.854 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:11.854 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:11.854 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:57:11.854 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:57:11.855 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:57:11.855 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:57:11.855 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:57:11.855 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:11.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.852+0000 7fd325394640 1 Processor -- start 2024-09-13T11:57:11.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.852+0000 7fd325394640 1 -- start start 2024-09-13T11:57:11.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.852+0000 7fd325394640 10 monclient: init 2024-09-13T11:57:11.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.852+0000 7fd325394640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:11.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.852+0000 7fd325394640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-13T11:57:11.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.852+0000 7fd325394640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd320137a60 addr v1:172.21.15.100:6789/0 2024-09-13T11:57:11.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.852+0000 7fd325394640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3200a2980 0x7fd32013c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:11.864 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.852+0000 7fd325394640 10 monclient(hunting): picked mon.noname-b con 0x7fd3200a2980 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:11.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.852+0000 7fd325394640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd32013ce90 0x7fd32013f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:11.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.852+0000 7fd325394640 10 monclient(hunting): picked mon.noname-a con 0x7fd32013ce90 addr v2:172.21.15.100:3300/0 2024-09-13T11:57:11.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.852+0000 7fd325394640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd320138550 con 0x7fd320137a60 2024-09-13T11:57:11.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.852+0000 7fd325394640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:11.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.852+0000 7fd325394640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7fd3200a4410 con 0x7fd32013ce90 2024-09-13T11:57:11.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.852+0000 7fd325394640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:11.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.852+0000 7fd325394640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fd3200a5be0 con 0x7fd3200a2980 2024-09-13T11:57:11.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.852+0000 7fd325394640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:11.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.852+0000 7fd325394640 10 monclient(hunting): authenticate will time out at 1012.446958s 2024-09-13T11:57:11.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd317fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3200a2980 0x7fd32013c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:11.865 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd31effd640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fd320137a60 0x7fd320137e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:47514/0 (socket says 172.21.15.100:47514) 2024-09-13T11:57:11.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd31effd640 1 -- 172.21.15.100:0/4049369219 learned_addr learned my addr 172.21.15.100:0/4049369219 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:57:11.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd317fff640 10 monclient(hunting): get_auth_request con 0x7fd3200a2980 auth_method 0 2024-09-13T11:57:11.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd317fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:11.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd317fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:11.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd317fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:11.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd31f7fe640 1 --2- 172.21.15.100:0/4049369219 >> v2:172.21.15.100:3300/0 conn(0x7fd32013ce90 0x7fd32013f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:11.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd31f7fe640 10 monclient(hunting): get_auth_request con 0x7fd32013ce90 auth_method 0 2024-09-13T11:57:11.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd31f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:11.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd31f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:11.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd31f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:11.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd317fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:11.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd317fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:11.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd317fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:11.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd31f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:11.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd31f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:11.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd31f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:11.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd317fff640 10 monclient(hunting): handle_auth_done global_id 24170 payload 274 2024-09-13T11:57:11.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd317fff640 1 -- 172.21.15.100:0/4049369219 >> v2:172.21.15.100:3300/0 conn(0x7fd32013ce90 msgr2=0x7fd32013f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:57:11.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd317fff640 1 --2- 172.21.15.100:0/4049369219 >> v2:172.21.15.100:3300/0 conn(0x7fd32013ce90 0x7fd32013f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:11.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd317fff640 1 -- 172.21.15.100:0/4049369219 >> v1:172.21.15.100:6789/0 conn(0x7fd320137a60 legacy=0x7fd320137e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:11.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd317fff640 10 monclient: _finish_hunting 0 2024-09-13T11:57:11.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd317fff640 1 monclient: found mon.noname-b 2024-09-13T11:57:11.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd317fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:11.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd317fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:57:11.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd317fff640 1 -- 172.21.15.100:0/4049369219 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3200a76b0 con 0x7fd3200a2980 2024-09-13T11:57:11.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.853+0000 7fd31f7fe640 1 --2- 172.21.15.100:0/4049369219 >> v2:172.21.15.100:3300/0 conn(0x7fd32013ce90 0x7fd32013f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:57:11.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd317fff640 1 --2- 172.21.15.100:0/4049369219 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3200a2980 0x7fd32013c950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd3080097a0 tx=0x7fd30802a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=315dd3ef489387ab server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:11.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd31e7fc640 1 -- 172.21.15.100:0/4049369219 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd308037070 con 0x7fd3200a2980 2024-09-13T11:57:11.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd31e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:11.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd31e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:57:11.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd31e7fc640 10 monclient: dump: 2024-09-13T11:57:11.868 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:11.868 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:11.868 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:11.868 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:11.868 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:11.868 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:11.868 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:11.869 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:11.869 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:11.869 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:11.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd31e7fc640 10 monclient: _finish_auth 0 2024-09-13T11:57:11.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd31e7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:57:11.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd31e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:11.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd31e7fc640 1 -- 172.21.15.100:0/4049369219 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd30802adb0 con 0x7fd3200a2980 2024-09-13T11:57:11.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd325394640 5 monclient: authenticate success, global_id 24170 2024-09-13T11:57:11.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd325394640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:57:11.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd31e7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:11.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd325394640 10 monclient: get_monmap_and_config success 2024-09-13T11:57:11.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd31e7fc640 1 -- 172.21.15.100:0/4049369219 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd308032660 con 0x7fd3200a2980 2024-09-13T11:57:11.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd31e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:11.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd325394640 4 set_mon_vals no callback set 2024-09-13T11:57:11.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd31e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:57:11.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd31e7fc640 10 monclient: dump: 2024-09-13T11:57:11.870 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:11.870 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:11.870 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:11.870 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:11.870 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:11.870 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:11.870 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:11.870 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:11.870 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:11.870 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:11.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd325394640 10 monclient: shutdown 2024-09-13T11:57:11.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd325394640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:11.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd325394640 1 -- 172.21.15.100:0/4049369219 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3200a2980 msgr2=0x7fd32013c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:11.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd325394640 1 --2- 172.21.15.100:0/4049369219 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3200a2980 0x7fd32013c950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd3080097a0 tx=0x7fd30802a060 comp rx=0 tx=0).stop 2024-09-13T11:57:11.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd325394640 1 -- 172.21.15.100:0/4049369219 shutdown_connections 2024-09-13T11:57:11.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd325394640 1 --2- 172.21.15.100:0/4049369219 >> v2:172.21.15.100:3300/0 conn(0x7fd32013ce90 0x7fd32013f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:11.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd325394640 1 --2- 172.21.15.100:0/4049369219 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd3200a2980 0x7fd32013c950 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:11.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.854+0000 7fd325394640 1 -- 172.21.15.100:0/4049369219 >> 172.21.15.100:0/4049369219 conn(0x7fd3200aa580 msgr2=0x7fd3200a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:11.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.855+0000 7fd325394640 1 -- 172.21.15.100:0/4049369219 shutdown_connections 2024-09-13T11:57:11.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.855+0000 7fd325394640 1 -- 172.21.15.100:0/4049369219 wait complete. 2024-09-13T11:57:11.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.855+0000 7fd325394640 10 monclient: build_initial_monmap 2024-09-13T11:57:11.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.855+0000 7fd325394640 10 monclient: monmap: 2024-09-13T11:57:11.872 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:11.872 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:57:11.872 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:57:11.872 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:57:11.872 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:11.872 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:11.872 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:57:11.872 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:57:11.872 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:57:11.872 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:11.872 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.855+0000 7fd325394640 1 Processor -- start 2024-09-13T11:57:11.872 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.855+0000 7fd325394640 1 -- start start 2024-09-13T11:57:11.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.855+0000 7fd325394640 10 monclient: init 2024-09-13T11:57:11.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd325394640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:11.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd325394640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:57:11.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd325394640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3200a2980 0x7fd3201367b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:11.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd325394640 10 monclient(hunting): picked mon.noname-b con 0x7fd3200a2980 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:57:11.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd325394640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd320137a60 0x7fd32009d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:11.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd325394640 10 monclient(hunting): picked mon.noname-c con 0x7fd320137a60 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:11.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd325394640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd32013ce90 0x7fd32009de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:11.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd325394640 10 monclient(hunting): picked mon.noname-a con 0x7fd32013ce90 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:57:11.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd325394640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:11.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd325394640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fd32006fd80 con 0x7fd32013ce90 2024-09-13T11:57:11.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd325394640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:11.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd325394640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fd3200a1700 con 0x7fd3200a2980 2024-09-13T11:57:11.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd325394640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:11.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd325394640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fd32006ff00 con 0x7fd320137a60 2024-09-13T11:57:11.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd325394640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:11.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd325394640 10 monclient(hunting): authenticate will time out at 1012.450380s 2024-09-13T11:57:11.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd31f7fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd32013ce90 0x7fd32009de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:11.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd317fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd320137a60 0x7fd32009d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:11.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd31effd640 1 -- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3200a2980 msgr2=0x7fd3201367b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.122:3300/0 2024-09-13T11:57:11.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd31effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3200a2980 0x7fd3201367b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:57:11.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd31f7fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd32013ce90 0x7fd32009de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55564/0 (socket says 172.21.15.100:55564) 2024-09-13T11:57:11.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd317fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd320137a60 0x7fd32009d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:46246/0 (socket says 172.21.15.100:46246) 2024-09-13T11:57:11.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd31f7fe640 1 -- 172.21.15.100:0/686034946 learned_addr learned my addr 172.21.15.100:0/686034946 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:11.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd31f7fe640 10 monclient(hunting): get_auth_request con 0x7fd32013ce90 auth_method 0 2024-09-13T11:57:11.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd31f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:11.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd31f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:11.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd31f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:11.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd317fff640 10 monclient(hunting): get_auth_request con 0x7fd320137a60 auth_method 0 2024-09-13T11:57:11.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd317fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:11.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd317fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:11.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd317fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:11.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd31f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:11.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd31f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:11.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd31f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:11.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd317fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:11.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd317fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:11.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd317fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:11.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd31f7fe640 10 monclient(hunting): handle_auth_done global_id 14946 payload 824 2024-09-13T11:57:11.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd31f7fe640 1 -- 172.21.15.100:0/686034946 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd320137a60 msgr2=0x7fd32009d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:11.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd31f7fe640 1 --2- 172.21.15.100:0/686034946 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd320137a60 0x7fd32009d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:11.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd31f7fe640 1 -- 172.21.15.100:0/686034946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3200a2980 msgr2=0x7fd3201367b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:57:11.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd31f7fe640 1 --2- 172.21.15.100:0/686034946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3200a2980 0x7fd3201367b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:11.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd31f7fe640 10 monclient: _finish_hunting 0 2024-09-13T11:57:11.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd31f7fe640 1 monclient: found mon.noname-a 2024-09-13T11:57:11.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd31f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:11.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd31f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:57:11.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd31f7fe640 1 -- 172.21.15.100:0/686034946 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3201c4b00 con 0x7fd32013ce90 2024-09-13T11:57:11.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.856+0000 7fd31f7fe640 1 --2- 172.21.15.100:0/686034946 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd32013ce90 0x7fd32009de40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd31800efc0 tx=0x7fd318009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:11.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.857+0000 7fd3177fe640 1 -- 172.21.15.100:0/686034946 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd318018030 con 0x7fd32013ce90 2024-09-13T11:57:11.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.857+0000 7fd3177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:11.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.857+0000 7fd3177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:57:11.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.857+0000 7fd3177fe640 10 monclient: dump: 2024-09-13T11:57:11.877 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:11.877 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:11.878 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:11.878 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:11.878 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:11.878 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:11.878 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:11.878 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:11.878 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:11.878 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:11.878 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.857+0000 7fd3177fe640 10 monclient: _finish_auth 0 2024-09-13T11:57:11.878 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.857+0000 7fd3177fe640 10 monclient: _check_auth_tickets 2024-09-13T11:57:11.878 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.857+0000 7fd3177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:11.878 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.857+0000 7fd3177fe640 1 -- 172.21.15.100:0/686034946 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd31800cde0 con 0x7fd32013ce90 2024-09-13T11:57:11.878 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.857+0000 7fd325394640 5 monclient: authenticate success, global_id 14946 2024-09-13T11:57:11.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.857+0000 7fd325394640 10 monclient: _renew_subs 2024-09-13T11:57:11.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.857+0000 7fd325394640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:11.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.857+0000 7fd325394640 1 -- 172.21.15.100:0/686034946 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd320120e60 con 0x7fd32013ce90 2024-09-13T11:57:11.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.857+0000 7fd3177fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:11.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.857+0000 7fd3177fe640 1 -- 172.21.15.100:0/686034946 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd318003bb0 con 0x7fd32013ce90 2024-09-13T11:57:11.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.857+0000 7fd3177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:11.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.857+0000 7fd3177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:11.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.857+0000 7fd3177fe640 10 monclient: dump: 2024-09-13T11:57:11.879 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:11.879 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:11.879 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:11.879 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:11.879 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:11.879 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:11.880 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:11.880 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:11.880 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:11.880 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:11.880 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.857+0000 7fd325394640 10 monclient: _renew_subs 2024-09-13T11:57:11.880 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.857+0000 7fd31dffb640 4 set_mon_vals no callback set 2024-09-13T11:57:11.880 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.857+0000 7fd325394640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:11.880 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.857+0000 7fd325394640 1 -- 172.21.15.100:0/686034946 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd32013ca40 con 0x7fd32013ce90 2024-09-13T11:57:11.880 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.858+0000 7fd3177fe640 1 -- 172.21.15.100:0/686034946 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fd318005d10 con 0x7fd32013ce90 2024-09-13T11:57:11.880 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.858+0000 7fd325394640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:11.880 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.858+0000 7fd325394640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:11.881 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.858+0000 7fd325394640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:11.881 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.858+0000 7fd325394640 1 -- 172.21.15.100:0/686034946 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd320137e40 con 0x7fd32013ce90 2024-09-13T11:57:11.881 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.858+0000 7fd3177fe640 1 --2- 172.21.15.100:0/686034946 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fd30003df80 0x7fd300040440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:11.881 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.858+0000 7fd3177fe640 1 -- 172.21.15.100:0/686034946 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(223..223 src has 1..223) ==== 4346+0+0 (secure 0 0 0) 0x7fd318053010 con 0x7fd32013ce90 2024-09-13T11:57:11.881 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.858+0000 7fd31effd640 1 --2- 172.21.15.100:0/686034946 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fd30003df80 0x7fd300040440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:11.881 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.858+0000 7fd31effd640 10 monclient: get_auth_request con 0x7fd30003df80 auth_method 0 2024-09-13T11:57:11.881 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.859+0000 7fd31effd640 1 --2- 172.21.15.100:0/686034946 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fd30003df80 0x7fd300040440 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd310007fd0 tx=0x7fd310007b40 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:11.882 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.862+0000 7fd3177fe640 1 -- 172.21.15.100:0/686034946 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fd318022070 con 0x7fd32013ce90 2024-09-13T11:57:11.882 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.862+0000 7fd3177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:11.882 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:11.862+0000 7fd3177fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:57:12.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:12.107+0000 7fd325394640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:57:12.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:12.107+0000 7fd325394640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:57:12.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:12.107+0000 7fd325394640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:12.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:12.107+0000 7fd325394640 1 -- 172.21.15.100:0/686034946 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd3201205a0 con 0x7fd32013ce90 2024-09-13T11:57:12.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:12.107+0000 7fd3177fe640 1 -- 172.21.15.100:0/686034946 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fd3201c4b00 con 0x7fd32013ce90 2024-09-13T11:57:12.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:12.107+0000 7fd3177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:57:12.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:12.107+0000 7fd3177fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:57:12.110 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:57:12.112 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:12.109+0000 7fd325394640 1 -- 172.21.15.100:0/686034946 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fd30003df80 msgr2=0x7fd300040440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:12.112 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:12.109+0000 7fd325394640 1 --2- 172.21.15.100:0/686034946 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fd30003df80 0x7fd300040440 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd310007fd0 tx=0x7fd310007b40 comp rx=0 tx=0).stop 2024-09-13T11:57:12.112 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:12.110+0000 7fd325394640 10 monclient: shutdown 2024-09-13T11:57:12.112 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:12.110+0000 7fd325394640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:12.112 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:12.110+0000 7fd325394640 1 -- 172.21.15.100:0/686034946 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd32013ce90 msgr2=0x7fd32009de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:12.113 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:12.110+0000 7fd325394640 1 --2- 172.21.15.100:0/686034946 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd32013ce90 0x7fd32009de40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd31800efc0 tx=0x7fd318009f90 comp rx=0 tx=0).stop 2024-09-13T11:57:12.113 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:12.110+0000 7fd325394640 1 -- 172.21.15.100:0/686034946 shutdown_connections 2024-09-13T11:57:12.113 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:12.110+0000 7fd325394640 1 --2- 172.21.15.100:0/686034946 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fd30003df80 0x7fd300040440 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:12.113 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:12.110+0000 7fd325394640 1 --2- 172.21.15.100:0/686034946 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd32013ce90 0x7fd32009de40 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:12.113 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:12.110+0000 7fd325394640 1 --2- 172.21.15.100:0/686034946 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd320137a60 0x7fd32009d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:12.113 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:12.110+0000 7fd325394640 1 --2- 172.21.15.100:0/686034946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3200a2980 0x7fd3201367b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:12.113 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:12.110+0000 7fd325394640 1 -- 172.21.15.100:0/686034946 >> 172.21.15.100:0/686034946 conn(0x7fd3200aa580 msgr2=0x7fd3200a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:12.113 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:12.110+0000 7fd325394640 1 -- 172.21.15.100:0/686034946 shutdown_connections 2024-09-13T11:57:12.113 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:12.110+0000 7fd325394640 1 -- 172.21.15.100:0/686034946 wait complete. 2024-09-13T11:57:15.124 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:57:15.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.230+0000 7f7667c40640 10 monclient: get_monmap_and_config 2024-09-13T11:57:15.232 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.230+0000 7f7667c40640 10 monclient: build_initial_monmap 2024-09-13T11:57:15.232 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.230+0000 7f7667c40640 10 monclient: monmap: 2024-09-13T11:57:15.232 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:15.232 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:15.232 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:57:15.231586+0000 2024-09-13T11:57:15.232 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:57:15.231586+0000 2024-09-13T11:57:15.232 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:15.232 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:15.232 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:57:15.232 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:57:15.232 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:57:15.233 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:57:15.233 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:57:15.233 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:15.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.231+0000 7f7667c40640 1 Processor -- start 2024-09-13T11:57:15.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.231+0000 7f7667c40640 1 -- start start 2024-09-13T11:57:15.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.231+0000 7f7667c40640 10 monclient: init 2024-09-13T11:57:15.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f7667c40640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:15.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f7667c40640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-13T11:57:15.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f7667c40640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f76600a15a0 0x7f76600a1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:15.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f7667c40640 10 monclient(hunting): picked mon.noname-a con 0x7f76600a15a0 addr v2:172.21.15.100:3300/0 2024-09-13T11:57:15.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f7667c40640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f766009be20 addr v1:172.21.15.100:6789/0 2024-09-13T11:57:15.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f7667c40640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76600a1ec0 0x7f766013c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:15.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f7667c40640 10 monclient(hunting): picked mon.noname-b con 0x7f76600a1ec0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:15.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f7667c40640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f766009c2a0 con 0x7f766009be20 2024-09-13T11:57:15.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f7667c40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:15.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f7667c40640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f766009dbe0 con 0x7f76600a15a0 2024-09-13T11:57:15.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f7667c40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:15.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f7667c40640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f766009d340 con 0x7f76600a1ec0 2024-09-13T11:57:15.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f7667c40640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:15.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f7667c40640 10 monclient(hunting): authenticate will time out at 1015.825151s 2024-09-13T11:57:15.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f76659dc640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f76600a15a0 0x7f76600a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:15.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f76659dc640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f76600a15a0 0x7f76600a1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55874/0 (socket says 172.21.15.100:55874) 2024-09-13T11:57:15.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f76659dc640 1 -- 172.21.15.100:0/3171376265 learned_addr learned my addr 172.21.15.100:0/3171376265 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:15.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f76659dc640 10 monclient(hunting): get_auth_request con 0x7f76600a15a0 auth_method 0 2024-09-13T11:57:15.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f76659dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:15.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f76659dc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:15.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f76659dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:15.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f76661dd640 1 --2- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76600a1ec0 0x7f766013c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:15.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f76661dd640 10 monclient(hunting): get_auth_request con 0x7f76600a1ec0 auth_method 0 2024-09-13T11:57:15.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f76661dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:15.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f76661dd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:15.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f76661dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:15.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f76659dc640 1 -- 172.21.15.100:0/3171376265 >> v2:172.21.15.100:3300/0 conn(0x7f76600a15a0 msgr2=0x7f76600a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:57:15.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f76659dc640 1 -- 172.21.15.100:0/3171376265 >> v2:172.21.15.100:3300/0 conn(0x7f76600a15a0 msgr2=0x7f76600a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:57:15.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f76659dc640 1 --2- 172.21.15.100:0/3171376265 >> v2:172.21.15.100:3300/0 conn(0x7f76600a15a0 0x7f76600a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:57:15.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f76661dd640 1 -- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76600a1ec0 msgr2=0x7f766013c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-13T11:57:15.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f76661dd640 1 -- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76600a1ec0 msgr2=0x7f766013c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:57:15.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f76661dd640 1 --2- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76600a1ec0 0x7f766013c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:57:15.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f76659dc640 1 --2- 172.21.15.100:0/3171376265 >> v2:172.21.15.100:3300/0 conn(0x7f76600a15a0 0x7f76600a1980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:57:15.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.232+0000 7f76661dd640 1 --2- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76600a1ec0 0x7f766013c270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:57:15.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.433+0000 7f76661dd640 1 --2- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76600a1ec0 0x7f766013c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:15.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.434+0000 7f76661dd640 10 monclient(hunting): get_auth_request con 0x7f76600a1ec0 auth_method 0 2024-09-13T11:57:15.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.433+0000 7f76659dc640 1 --2- 172.21.15.100:0/3171376265 >> v2:172.21.15.100:3300/0 conn(0x7f76600a15a0 0x7f76600a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:15.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.434+0000 7f76661dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:15.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.434+0000 7f76661dd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:15.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.434+0000 7f76661dd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:57:15.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.434+0000 7f76659dc640 10 monclient(hunting): get_auth_request con 0x7f76600a15a0 auth_method 0 2024-09-13T11:57:15.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.434+0000 7f76659dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:15.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.434+0000 7f76659dc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:15.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.434+0000 7f76659dc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:57:15.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.434+0000 7f76661dd640 1 -- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76600a1ec0 msgr2=0x7f766013c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:57:15.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.434+0000 7f76661dd640 1 -- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76600a1ec0 msgr2=0x7f766013c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:57:15.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.434+0000 7f76661dd640 1 --2- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76600a1ec0 0x7f766013c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:57:15.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.434+0000 7f76661dd640 1 --2- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76600a1ec0 0x7f766013c270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T11:57:15.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.434+0000 7f76659dc640 1 -- 172.21.15.100:0/3171376265 >> v2:172.21.15.100:3300/0 conn(0x7f76600a15a0 msgr2=0x7f76600a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-13T11:57:15.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.434+0000 7f76659dc640 1 -- 172.21.15.100:0/3171376265 >> v2:172.21.15.100:3300/0 conn(0x7f76600a15a0 msgr2=0x7f76600a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:57:15.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.434+0000 7f76659dc640 1 --2- 172.21.15.100:0/3171376265 >> v2:172.21.15.100:3300/0 conn(0x7f76600a15a0 0x7f76600a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:57:15.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.434+0000 7f76659dc640 1 --2- 172.21.15.100:0/3171376265 >> v2:172.21.15.100:3300/0 conn(0x7f76600a15a0 0x7f76600a1980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T11:57:15.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.835+0000 7f76661dd640 1 --2- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76600a1ec0 0x7f766013c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:15.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.835+0000 7f76659dc640 1 --2- 172.21.15.100:0/3171376265 >> v2:172.21.15.100:3300/0 conn(0x7f76600a15a0 0x7f76600a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:15.837 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.836+0000 7f76661dd640 10 monclient(hunting): get_auth_request con 0x7f76600a1ec0 auth_method 0 2024-09-13T11:57:15.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.836+0000 7f76661dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:15.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.836+0000 7f76661dd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:15.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.836+0000 7f76661dd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:57:15.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.836+0000 7f76659dc640 10 monclient(hunting): get_auth_request con 0x7f76600a15a0 auth_method 0 2024-09-13T11:57:15.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.836+0000 7f76659dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:15.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.836+0000 7f76659dc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:15.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.836+0000 7f76659dc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:57:15.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.836+0000 7f76661dd640 1 -- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76600a1ec0 msgr2=0x7f766013c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:57:15.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.836+0000 7f76661dd640 1 -- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76600a1ec0 msgr2=0x7f766013c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:57:15.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.836+0000 7f76661dd640 1 --2- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76600a1ec0 0x7f766013c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:57:15.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.836+0000 7f76659dc640 1 -- 172.21.15.100:0/3171376265 >> v2:172.21.15.100:3300/0 conn(0x7f76600a15a0 msgr2=0x7f76600a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-13T11:57:15.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.836+0000 7f76659dc640 1 -- 172.21.15.100:0/3171376265 >> v2:172.21.15.100:3300/0 conn(0x7f76600a15a0 msgr2=0x7f76600a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:57:15.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.836+0000 7f76659dc640 1 --2- 172.21.15.100:0/3171376265 >> v2:172.21.15.100:3300/0 conn(0x7f76600a15a0 0x7f76600a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:57:15.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.836+0000 7f76661dd640 1 --2- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76600a1ec0 0x7f766013c270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T11:57:15.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:15.836+0000 7f76659dc640 1 --2- 172.21.15.100:0/3171376265 >> v2:172.21.15.100:3300/0 conn(0x7f76600a15a0 0x7f76600a1980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T11:57:16.638 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:16.637+0000 7f76659dc640 1 --2- 172.21.15.100:0/3171376265 >> v2:172.21.15.100:3300/0 conn(0x7f76600a15a0 0x7f76600a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:16.639 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:16.637+0000 7f76661dd640 1 --2- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76600a1ec0 0x7f766013c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:16.639 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:16.637+0000 7f76661dd640 10 monclient(hunting): get_auth_request con 0x7f76600a1ec0 auth_method 0 2024-09-13T11:57:16.639 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:16.637+0000 7f76661dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:16.639 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:16.637+0000 7f76661dd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:16.639 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:16.637+0000 7f76661dd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:57:16.639 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:16.637+0000 7f76659dc640 10 monclient(hunting): get_auth_request con 0x7f76600a15a0 auth_method 0 2024-09-13T11:57:16.639 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:16.637+0000 7f76659dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:16.639 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:16.637+0000 7f76659dc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:16.639 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:16.637+0000 7f76659dc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:57:16.639 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:16.637+0000 7f76661dd640 1 -- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76600a1ec0 msgr2=0x7f766013c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:57:16.640 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:16.637+0000 7f76661dd640 1 -- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76600a1ec0 msgr2=0x7f766013c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:57:16.640 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:16.637+0000 7f76659dc640 1 -- 172.21.15.100:0/3171376265 >> v2:172.21.15.100:3300/0 conn(0x7f76600a15a0 msgr2=0x7f76600a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-13T11:57:16.640 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:16.637+0000 7f76661dd640 1 --2- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76600a1ec0 0x7f766013c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:57:16.640 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:16.637+0000 7f76659dc640 1 -- 172.21.15.100:0/3171376265 >> v2:172.21.15.100:3300/0 conn(0x7f76600a15a0 msgr2=0x7f76600a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:57:16.640 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:16.637+0000 7f76659dc640 1 --2- 172.21.15.100:0/3171376265 >> v2:172.21.15.100:3300/0 conn(0x7f76600a15a0 0x7f76600a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:57:16.640 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:16.637+0000 7f76661dd640 1 --2- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76600a1ec0 0x7f766013c270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T11:57:16.640 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:16.637+0000 7f76659dc640 1 --2- 172.21.15.100:0/3171376265 >> v2:172.21.15.100:3300/0 conn(0x7f76600a15a0 0x7f76600a1980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T11:57:18.234 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.232+0000 7f76537fe640 10 monclient(hunting): tick 2024-09-13T11:57:18.234 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.232+0000 7f76537fe640 10 monclient(hunting): _check_auth_tickets 2024-09-13T11:57:18.234 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.232+0000 7f76537fe640 1 monclient(hunting): continuing hunt 2024-09-13T11:57:18.234 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.232+0000 7f76537fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-13T11:57:18.234 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.232+0000 7f76537fe640 1 -- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76600a1ec0 msgr2=0x7f766013c270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:57:18.234 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.232+0000 7f76537fe640 1 --2- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76600a1ec0 0x7f766013c270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:18.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.232+0000 7f76537fe640 1 -- 172.21.15.100:0/3171376265 >> v2:172.21.15.100:3300/0 conn(0x7f76600a15a0 msgr2=0x7f76600a1980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:57:18.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.232+0000 7f76537fe640 1 --2- 172.21.15.100:0/3171376265 >> v2:172.21.15.100:3300/0 conn(0x7f76600a15a0 0x7f76600a1980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:18.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.232+0000 7f76537fe640 1 -- 172.21.15.100:0/3171376265 >> v1:172.21.15.100:6789/0 conn(0x7f766009be20 legacy=0x7f766013bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:18.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.232+0000 7f76537fe640 10 monclient: _add_conns ranks=[4,1] 2024-09-13T11:57:18.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.232+0000 7f76537fe640 1 --2- 172.21.15.100:0/3171376265 >> v2:172.21.15.122:3300/0 conn(0x7f76440036a0 0x7f7644005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:18.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.232+0000 7f76537fe640 10 monclient(hunting): picked mon.noname-c con 0x7f76440036a0 addr v2:172.21.15.122:3300/0 2024-09-13T11:57:18.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.232+0000 7f76537fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7644006170 addr v1:172.21.15.122:6789/0 2024-09-13T11:57:18.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.232+0000 7f76537fe640 1 -- 172.21.15.100:0/3171376265 --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f766009fed0 con 0x7f7644006170 2024-09-13T11:57:18.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.232+0000 7f76537fe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:18.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.232+0000 7f76537fe640 1 -- 172.21.15.100:0/3171376265 --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f766009dbe0 con 0x7f76440036a0 2024-09-13T11:57:18.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.232+0000 7f76537fe640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:18.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.232+0000 7f76659dc640 1 -- 172.21.15.100:0/3171376265 >> v2:172.21.15.122:3300/0 conn(0x7f76440036a0 msgr2=0x7f7644005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.122:3300/0 2024-09-13T11:57:18.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.232+0000 7f76651db640 1 -- 172.21.15.100:0/3171376265 >> v1:172.21.15.122:6789/0 conn(0x7f7644006170 legacy=0x7f7644008630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6789/0 2024-09-13T11:57:18.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.232+0000 7f76659dc640 1 --2- 172.21.15.100:0/3171376265 >> v2:172.21.15.122:3300/0 conn(0x7f76440036a0 0x7f7644005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:57:18.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.433+0000 7f76651db640 1 -- 172.21.15.100:0/3171376265 >> v1:172.21.15.122:6789/0 conn(0x7f7644006170 legacy=0x7f7644008630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6789/0 2024-09-13T11:57:18.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.433+0000 7f76659dc640 1 -- 172.21.15.100:0/3171376265 >> v2:172.21.15.122:3300/0 conn(0x7f76440036a0 msgr2=0x7f7644005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.122:3300/0 2024-09-13T11:57:18.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.433+0000 7f76659dc640 1 --2- 172.21.15.100:0/3171376265 >> v2:172.21.15.122:3300/0 conn(0x7f76440036a0 0x7f7644005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T11:57:18.836 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.834+0000 7f76659dc640 1 -- 172.21.15.100:0/3171376265 >> v2:172.21.15.122:3300/0 conn(0x7f76440036a0 msgr2=0x7f7644005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.122:3300/0 2024-09-13T11:57:18.836 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.834+0000 7f76651db640 1 -- 172.21.15.100:0/3171376265 >> v1:172.21.15.122:6789/0 conn(0x7f7644006170 legacy=0x7f7644008630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6789/0 2024-09-13T11:57:18.836 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:18.834+0000 7f76659dc640 1 --2- 172.21.15.100:0/3171376265 >> v2:172.21.15.122:3300/0 conn(0x7f76440036a0 0x7f7644005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T11:57:19.637 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:19.636+0000 7f76651db640 1 -- 172.21.15.100:0/3171376265 >> v1:172.21.15.122:6789/0 conn(0x7f7644006170 legacy=0x7f7644008630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6789/0 2024-09-13T11:57:19.637 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:19.636+0000 7f76659dc640 1 -- 172.21.15.100:0/3171376265 >> v2:172.21.15.122:3300/0 conn(0x7f76440036a0 msgr2=0x7f7644005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.122:3300/0 2024-09-13T11:57:19.637 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:19.636+0000 7f76659dc640 1 --2- 172.21.15.100:0/3171376265 >> v2:172.21.15.122:3300/0 conn(0x7f76440036a0 0x7f7644005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T11:57:20.209 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38118]: starting. 2024-09-13T11:57:20.210 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38118]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-38117 2024-09-13T11:57:20.210 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38118]: created object 0... 2024-09-13T11:57:20.210 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38118]: created object 25... 2024-09-13T11:57:20.210 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38118]: created object 49... 2024-09-13T11:57:20.210 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38118]: finishing. 2024-09-13T11:57:20.210 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38118]: shutting down. 2024-09-13T11:57:21.132 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38119]: starting. 2024-09-13T11:57:21.132 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38119]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-38117 2024-09-13T11:57:21.132 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38119]: shutting down. 2024-09-13T11:57:21.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.232+0000 7f76537fe640 10 monclient(hunting): tick 2024-09-13T11:57:21.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.232+0000 7f76537fe640 10 monclient(hunting): _check_auth_tickets 2024-09-13T11:57:21.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.232+0000 7f76537fe640 1 monclient(hunting): continuing hunt 2024-09-13T11:57:21.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.232+0000 7f76537fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-13T11:57:21.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.232+0000 7f76537fe640 1 -- 172.21.15.100:0/3171376265 >> v2:172.21.15.122:3300/0 conn(0x7f76440036a0 msgr2=0x7f7644005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:57:21.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.232+0000 7f76537fe640 1 --2- 172.21.15.100:0/3171376265 >> v2:172.21.15.122:3300/0 conn(0x7f76440036a0 0x7f7644005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:21.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.232+0000 7f76537fe640 1 -- 172.21.15.100:0/3171376265 >> v1:172.21.15.122:6789/0 conn(0x7f7644006170 legacy=0x7f7644008630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:57:21.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.232+0000 7f76661dd640 1 -- 172.21.15.100:0/3171376265 reap_dead start 2024-09-13T11:57:21.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.232+0000 7f76537fe640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-13T11:57:21.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.232+0000 7f76537fe640 1 --2- 172.21.15.100:0/3171376265 >> v2:172.21.15.100:3300/0 conn(0x7f7644009eb0 0x7f7644005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:21.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.232+0000 7f76537fe640 10 monclient(hunting): picked mon.noname-a con 0x7f7644009eb0 addr v2:172.21.15.100:3300/0 2024-09-13T11:57:21.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.232+0000 7f76537fe640 1 --2- 172.21.15.100:0/3171376265 >> v2:172.21.15.122:3300/0 conn(0x7f764400a350 0x7f764400a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:21.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.232+0000 7f76537fe640 10 monclient(hunting): picked mon.noname-c con 0x7f764400a350 addr v2:172.21.15.122:3300/0 2024-09-13T11:57:21.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.232+0000 7f76537fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f76440085d0 addr v1:172.21.15.100:6789/0 2024-09-13T11:57:21.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.232+0000 7f76537fe640 1 -- 172.21.15.100:0/3171376265 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f764400aed0 con 0x7f76440085d0 2024-09-13T11:57:21.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.232+0000 7f76537fe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:21.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.232+0000 7f76537fe640 1 -- 172.21.15.100:0/3171376265 --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f766009dbe0 con 0x7f7644009eb0 2024-09-13T11:57:21.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.232+0000 7f76537fe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:21.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.232+0000 7f76537fe640 1 -- 172.21.15.100:0/3171376265 --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f766009d340 con 0x7f764400a350 2024-09-13T11:57:21.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.232+0000 7f76537fe640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:21.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.232+0000 7f76651db640 1 -- 172.21.15.100:0/3171376265 >> v2:172.21.15.122:3300/0 conn(0x7f764400a350 msgr2=0x7f764400a800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.122:3300/0 2024-09-13T11:57:21.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.233+0000 7f76651db640 1 --2- 172.21.15.100:0/3171376265 >> v2:172.21.15.122:3300/0 conn(0x7f764400a350 0x7f764400a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:57:21.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.233+0000 7f76659dc640 1 --2- 172.21.15.100:0/3171376265 >> v2:172.21.15.100:3300/0 conn(0x7f7644009eb0 0x7f7644005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:21.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.233+0000 7f76659dc640 10 monclient(hunting): get_auth_request con 0x7f7644009eb0 auth_method 0 2024-09-13T11:57:21.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.233+0000 7f76659dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:21.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.233+0000 7f76659dc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:21.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.233+0000 7f76659dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:21.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.233+0000 7f76659dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:21.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.233+0000 7f76659dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:21.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.233+0000 7f76659dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:21.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.233+0000 7f76649da640 1 -- 172.21.15.100:0/3171376265 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1485217568 0 0) 0x7f764400aed0 con 0x7f76440085d0 2024-09-13T11:57:21.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.233+0000 7f76649da640 10 monclient(hunting): _negotiate 2024-09-13T11:57:21.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.233+0000 7f76649da640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:21.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.233+0000 7f76649da640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:21.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.233+0000 7f76649da640 10 monclient(hunting): my global_id is 14973 2024-09-13T11:57:21.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.233+0000 7f76649da640 1 -- 172.21.15.100:0/3171376265 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f764c003520 con 0x7f76440085d0 2024-09-13T11:57:21.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.233+0000 7f76659dc640 10 monclient(hunting): handle_auth_done global_id 14970 payload 274 2024-09-13T11:57:21.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.233+0000 7f76659dc640 1 -- 172.21.15.100:0/3171376265 >> v2:172.21.15.122:3300/0 conn(0x7f764400a350 msgr2=0x7f764400a800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:57:21.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.233+0000 7f76659dc640 1 --2- 172.21.15.100:0/3171376265 >> v2:172.21.15.122:3300/0 conn(0x7f764400a350 0x7f764400a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:21.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.233+0000 7f76659dc640 1 -- 172.21.15.100:0/3171376265 >> v1:172.21.15.100:6789/0 conn(0x7f76440085d0 legacy=0x7f7644008d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:21.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.233+0000 7f76659dc640 10 monclient: _finish_hunting 0 2024-09-13T11:57:21.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.233+0000 7f76659dc640 1 monclient: found mon.noname-a 2024-09-13T11:57:21.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.233+0000 7f76659dc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:21.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.233+0000 7f76659dc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:57:21.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.233+0000 7f76659dc640 1 -- 172.21.15.100:0/3171376265 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f766009fed0 con 0x7f7644009eb0 2024-09-13T11:57:21.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.234+0000 7f76659dc640 1 --2- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7644009eb0 0x7f7644005b20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7644003650 tx=0x7f765c02af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77777d2aad98f238 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:21.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.234+0000 7f76649da640 1 -- 172.21.15.100:0/3171376265 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f765c031070 con 0x7f7644009eb0 2024-09-13T11:57:21.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.234+0000 7f76649da640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:21.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.234+0000 7f76649da640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:57:21.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.234+0000 7f76649da640 10 monclient: dump: 2024-09-13T11:57:21.240 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:21.241 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:21.241 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.241 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.241 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:21.241 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:21.241 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:21.241 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:21.241 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:21.241 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:21.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.234+0000 7f76649da640 10 monclient: _finish_auth 0 2024-09-13T11:57:21.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.234+0000 7f76649da640 10 monclient: _check_auth_tickets 2024-09-13T11:57:21.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.234+0000 7f76649da640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:21.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.234+0000 7f76649da640 1 -- 172.21.15.100:0/3171376265 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f765c02bc80 con 0x7f7644009eb0 2024-09-13T11:57:21.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.234+0000 7f7667c40640 5 monclient: authenticate success, global_id 14970 2024-09-13T11:57:21.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.234+0000 7f7667c40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:57:21.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.234+0000 7f76649da640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:21.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.234+0000 7f7667c40640 10 monclient: get_monmap_and_config success 2024-09-13T11:57:21.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.234+0000 7f76649da640 1 -- 172.21.15.100:0/3171376265 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f765c00f040 con 0x7f7644009eb0 2024-09-13T11:57:21.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.234+0000 7f7667c40640 4 set_mon_vals no callback set 2024-09-13T11:57:21.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.234+0000 7f76649da640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:21.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.234+0000 7f76649da640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:21.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.234+0000 7f76649da640 10 monclient: dump: 2024-09-13T11:57:21.242 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:21.242 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:21.243 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.243 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.243 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:21.243 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:21.243 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:21.243 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:21.243 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:21.243 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:21.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.235+0000 7f7667c40640 10 monclient: shutdown 2024-09-13T11:57:21.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.235+0000 7f7667c40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:21.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.235+0000 7f7667c40640 1 -- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7644009eb0 msgr2=0x7f7644005b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:21.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.235+0000 7f7667c40640 1 --2- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7644009eb0 0x7f7644005b20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7644003650 tx=0x7f765c02af70 comp rx=0 tx=0).stop 2024-09-13T11:57:21.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.235+0000 7f7667c40640 1 -- 172.21.15.100:0/3171376265 shutdown_connections 2024-09-13T11:57:21.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.235+0000 7f7667c40640 1 --2- 172.21.15.100:0/3171376265 >> v2:172.21.15.122:3300/0 conn(0x7f764400a350 0x7f764400a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:21.244 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.235+0000 7f7667c40640 1 --2- 172.21.15.100:0/3171376265 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7644009eb0 0x7f7644005b20 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:21.244 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.235+0000 7f7667c40640 1 -- 172.21.15.100:0/3171376265 >> 172.21.15.100:0/3171376265 conn(0x7f76600a6470 msgr2=0x7f76600a2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:21.244 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.235+0000 7f7667c40640 1 -- 172.21.15.100:0/3171376265 shutdown_connections 2024-09-13T11:57:21.244 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.235+0000 7f7667c40640 1 -- 172.21.15.100:0/3171376265 wait complete. 2024-09-13T11:57:21.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.236+0000 7f7667c40640 10 monclient: build_initial_monmap 2024-09-13T11:57:21.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.236+0000 7f7667c40640 10 monclient: monmap: 2024-09-13T11:57:21.248 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:21.248 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:57:21.248 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:57:21.248 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:57:21.248 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:21.248 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:21.249 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:57:21.249 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:57:21.249 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:57:21.249 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:21.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.236+0000 7f7667c40640 1 Processor -- start 2024-09-13T11:57:21.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.236+0000 7f7667c40640 1 -- start start 2024-09-13T11:57:21.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.236+0000 7f7667c40640 10 monclient: init 2024-09-13T11:57:21.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f7667c40640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:21.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f7667c40640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:57:21.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f7667c40640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76440085d0 0x7f76601d0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:21.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f7667c40640 10 monclient(hunting): picked mon.noname-c con 0x7f76440085d0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:21.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f7667c40640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7644009eb0 0x7f766013bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:21.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f7667c40640 10 monclient(hunting): picked mon.noname-a con 0x7f7644009eb0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:57:21.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f7667c40640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f764400a350 0x7f766013c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:21.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f7667c40640 10 monclient(hunting): picked mon.noname-b con 0x7f764400a350 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:57:21.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f7667c40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:21.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f7667c40640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f76600a3250 con 0x7f7644009eb0 2024-09-13T11:57:21.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f7667c40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:21.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f7667c40640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f766009e5f0 con 0x7f764400a350 2024-09-13T11:57:21.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f7667c40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:21.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f7667c40640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f76600a33d0 con 0x7f76440085d0 2024-09-13T11:57:21.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f7667c40640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:21.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f7667c40640 10 monclient(hunting): authenticate will time out at 1021.830351s 2024-09-13T11:57:21.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76659dc640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76440085d0 0x7f76601d0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:21.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76651db640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7644009eb0 0x7f766013bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:21.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76661dd640 1 -- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f764400a350 msgr2=0x7f766013c0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.122:3300/0 2024-09-13T11:57:21.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76661dd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f764400a350 0x7f766013c0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:57:21.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76659dc640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76440085d0 0x7f76601d0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:54918/0 (socket says 172.21.15.100:54918) 2024-09-13T11:57:21.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76659dc640 1 -- 172.21.15.100:0/1013963409 learned_addr learned my addr 172.21.15.100:0/1013963409 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:21.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76651db640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7644009eb0 0x7f766013bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55926/0 (socket says 172.21.15.100:55926) 2024-09-13T11:57:21.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76659dc640 10 monclient(hunting): get_auth_request con 0x7f76440085d0 auth_method 0 2024-09-13T11:57:21.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76659dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:21.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76659dc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:21.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76659dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:21.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76651db640 10 monclient(hunting): get_auth_request con 0x7f7644009eb0 auth_method 0 2024-09-13T11:57:21.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76651db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:21.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76651db640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:21.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76651db640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:21.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76659dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:21.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76659dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:21.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76659dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:21.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76651db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:21.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76651db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:21.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76651db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:21.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76651db640 10 monclient(hunting): handle_auth_done global_id 14976 payload 824 2024-09-13T11:57:21.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76651db640 1 -- 172.21.15.100:0/1013963409 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76440085d0 msgr2=0x7f76601d0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:21.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76651db640 1 --2- 172.21.15.100:0/1013963409 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76440085d0 0x7f76601d0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:21.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76651db640 1 -- 172.21.15.100:0/1013963409 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f764400a350 msgr2=0x7f766013c0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:57:21.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76651db640 1 --2- 172.21.15.100:0/1013963409 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f764400a350 0x7f766013c0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:21.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76651db640 10 monclient: _finish_hunting 0 2024-09-13T11:57:21.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76651db640 1 monclient: found mon.noname-a 2024-09-13T11:57:21.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76651db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:21.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76651db640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:57:21.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76651db640 1 -- 172.21.15.100:0/1013963409 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7660076a90 con 0x7f7644009eb0 2024-09-13T11:57:21.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.237+0000 7f76659dc640 1 --2- 172.21.15.100:0/1013963409 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76440085d0 0x7f76601d0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:57:21.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.238+0000 7f76651db640 1 --2- 172.21.15.100:0/1013963409 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7644009eb0 0x7f766013bbb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f765400c420 tx=0x7f765400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:21.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.238+0000 7f76527fc640 1 -- 172.21.15.100:0/1013963409 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7654017070 con 0x7f7644009eb0 2024-09-13T11:57:21.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.238+0000 7f76527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:21.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.238+0000 7f76527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:57:21.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.238+0000 7f76527fc640 10 monclient: dump: 2024-09-13T11:57:21.253 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:21.253 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:21.253 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.254 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.254 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:21.254 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:21.254 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:21.254 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:21.254 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:21.254 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:21.254 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.238+0000 7f76527fc640 10 monclient: _finish_auth 0 2024-09-13T11:57:21.254 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.238+0000 7f76527fc640 10 monclient: _check_auth_tickets 2024-09-13T11:57:21.254 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.238+0000 7f76527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:21.254 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.238+0000 7f7667c40640 5 monclient: authenticate success, global_id 14976 2024-09-13T11:57:21.254 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.238+0000 7f76527fc640 1 -- 172.21.15.100:0/1013963409 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76540074e0 con 0x7f7644009eb0 2024-09-13T11:57:21.254 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.238+0000 7f7667c40640 10 monclient: _renew_subs 2024-09-13T11:57:21.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.238+0000 7f7667c40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:21.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.238+0000 7f7667c40640 1 -- 172.21.15.100:0/1013963409 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f766012d5b0 con 0x7f7644009eb0 2024-09-13T11:57:21.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.238+0000 7f76527fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:21.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.238+0000 7f76527fc640 1 -- 172.21.15.100:0/1013963409 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7654005a00 con 0x7f7644009eb0 2024-09-13T11:57:21.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.238+0000 7f76527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:21.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.238+0000 7f76527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:21.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.238+0000 7f76527fc640 10 monclient: dump: 2024-09-13T11:57:21.255 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:21.255 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:21.255 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.255 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.255 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:21.255 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:21.256 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:21.256 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:21.256 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:21.256 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:21.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.238+0000 7f7667c40640 10 monclient: _renew_subs 2024-09-13T11:57:21.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.238+0000 7f7667c40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:21.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.238+0000 7f7667c40640 1 -- 172.21.15.100:0/1013963409 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f766012b250 con 0x7f7644009eb0 2024-09-13T11:57:21.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.238+0000 7f7653fff640 4 set_mon_vals no callback set 2024-09-13T11:57:21.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.239+0000 7f76527fc640 1 -- 172.21.15.100:0/1013963409 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f765401a070 con 0x7f7644009eb0 2024-09-13T11:57:21.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.239+0000 7f7667c40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:21.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.239+0000 7f7667c40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:21.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.239+0000 7f7667c40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:21.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.239+0000 7f7667c40640 1 -- 172.21.15.100:0/1013963409 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7660073f00 con 0x7f7644009eb0 2024-09-13T11:57:21.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.239+0000 7f76527fc640 1 --2- 172.21.15.100:0/1013963409 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f764403f860 0x7f7644041d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:21.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.239+0000 7f76527fc640 1 -- 172.21.15.100:0/1013963409 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 4346+0+0 (secure 0 0 0) 0x7f7654051a60 con 0x7f7644009eb0 2024-09-13T11:57:21.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.240+0000 7f76659dc640 1 --2- 172.21.15.100:0/1013963409 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f764403f860 0x7f7644041d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:21.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.240+0000 7f76659dc640 10 monclient: get_auth_request con 0x7f764403f860 auth_method 0 2024-09-13T11:57:21.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.240+0000 7f76659dc640 1 --2- 172.21.15.100:0/1013963409 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f764403f860 0x7f7644041d20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f765c000c00 tx=0x7f765c010040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:21.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.244+0000 7f76527fc640 1 -- 172.21.15.100:0/1013963409 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f7660073f00 con 0x7f7644009eb0 2024-09-13T11:57:21.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.244+0000 7f76527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:21.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.244+0000 7f76527fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:57:21.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.489+0000 7f7667c40640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:57:21.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.489+0000 7f7667c40640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:57:21.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.489+0000 7f7667c40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:21.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.489+0000 7f7667c40640 1 -- 172.21.15.100:0/1013963409 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f76600a1a10 con 0x7f7644009eb0 2024-09-13T11:57:21.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.489+0000 7f76527fc640 1 -- 172.21.15.100:0/1013963409 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f7660076a90 con 0x7f7644009eb0 2024-09-13T11:57:21.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.489+0000 7f76527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:57:21.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.489+0000 7f76527fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:57:21.491 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":26,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:57:21.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.491+0000 7f7667c40640 1 -- 172.21.15.100:0/1013963409 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f764403f860 msgr2=0x7f7644041d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:21.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.491+0000 7f7667c40640 1 --2- 172.21.15.100:0/1013963409 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f764403f860 0x7f7644041d20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f765c000c00 tx=0x7f765c010040 comp rx=0 tx=0).stop 2024-09-13T11:57:21.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.491+0000 7f7667c40640 10 monclient: shutdown 2024-09-13T11:57:21.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.491+0000 7f7667c40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:21.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.491+0000 7f7667c40640 1 -- 172.21.15.100:0/1013963409 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7644009eb0 msgr2=0x7f766013bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:21.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.491+0000 7f7667c40640 1 --2- 172.21.15.100:0/1013963409 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7644009eb0 0x7f766013bbb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f765400c420 tx=0x7f765400c8e0 comp rx=0 tx=0).stop 2024-09-13T11:57:21.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.491+0000 7f7667c40640 1 -- 172.21.15.100:0/1013963409 shutdown_connections 2024-09-13T11:57:21.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.491+0000 7f7667c40640 1 --2- 172.21.15.100:0/1013963409 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f764403f860 0x7f7644041d20 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:21.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.491+0000 7f7667c40640 1 --2- 172.21.15.100:0/1013963409 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7644009eb0 0x7f766013bbb0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:21.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.491+0000 7f7667c40640 1 --2- 172.21.15.100:0/1013963409 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f764400a350 0x7f766013c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:21.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.491+0000 7f7667c40640 1 --2- 172.21.15.100:0/1013963409 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76440085d0 0x7f76601d0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:21.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.491+0000 7f7667c40640 1 -- 172.21.15.100:0/1013963409 >> 172.21.15.100:0/1013963409 conn(0x7f76600a6470 msgr2=0x7f7660071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:21.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.491+0000 7f7667c40640 1 -- 172.21.15.100:0/1013963409 shutdown_connections 2024-09-13T11:57:21.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.492+0000 7f7667c40640 1 -- 172.21.15.100:0/1013963409 wait complete. 2024-09-13T11:57:21.503 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-13T11:57:21.504 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-13T11:57:21.504 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-13T11:57:21.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.600+0000 7f3d311ca640 10 monclient: get_monmap_and_config 2024-09-13T11:57:21.601 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.600+0000 7f3d311ca640 10 monclient: build_initial_monmap 2024-09-13T11:57:21.602 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.600+0000 7f3d311ca640 10 monclient: monmap: 2024-09-13T11:57:21.602 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:21.602 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:21.602 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:57:21.600991+0000 2024-09-13T11:57:21.602 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:57:21.600991+0000 2024-09-13T11:57:21.602 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:21.602 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:21.602 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:57:21.602 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:57:21.602 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:57:21.602 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:57:21.602 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:57:21.602 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:21.603 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.600+0000 7f3d311ca640 1 Processor -- start 2024-09-13T11:57:21.603 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.600+0000 7f3d311ca640 1 -- start start 2024-09-13T11:57:21.603 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.600+0000 7f3d311ca640 10 monclient: init 2024-09-13T11:57:21.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d311ca640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:21.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d311ca640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-13T11:57:21.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d311ca640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3d2c06ea50 addr v1:172.21.15.100:6789/0 2024-09-13T11:57:21.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d311ca640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3d2c0a2860 addr v1:172.21.15.122:6789/0 2024-09-13T11:57:21.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d311ca640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f3d2c13da00 0x7f3d2c13fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:21.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d311ca640 10 monclient(hunting): picked mon.noname-a con 0x7f3d2c13da00 addr v2:172.21.15.100:3300/0 2024-09-13T11:57:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d311ca640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d2c0a1590 con 0x7f3d2c06ea50 2024-09-13T11:57:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d311ca640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d2c0a1870 con 0x7f3d2c0a2860 2024-09-13T11:57:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d311ca640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d311ca640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f3d2c0a2cb0 con 0x7f3d2c13da00 2024-09-13T11:57:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d311ca640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d311ca640 10 monclient(hunting): authenticate will time out at 1022.194249s 2024-09-13T11:57:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d2b59e640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f3d2c13da00 0x7f3d2c13fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d2a59c640 1 -- >> v1:172.21.15.122:6789/0 conn(0x7f3d2c0a2860 legacy=0x7f3d2c06f540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6789/0 2024-09-13T11:57:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d2ad9d640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f3d2c06ea50 0x7f3d2c06ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:37326/0 (socket says 172.21.15.100:37326) 2024-09-13T11:57:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d2b59e640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f3d2c13da00 0x7f3d2c13fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55934/0 (socket says 172.21.15.100:55934) 2024-09-13T11:57:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d2ad9d640 1 -- 172.21.15.100:0/3844058001 learned_addr learned my addr 172.21.15.100:0/3844058001 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:57:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d2b59e640 10 monclient(hunting): get_auth_request con 0x7f3d2c13da00 auth_method 0 2024-09-13T11:57:21.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d2b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:21.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d2b59e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:21.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d2b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:21.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d2b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:21.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d2b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:21.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d2b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:21.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d2b59e640 10 monclient(hunting): handle_auth_done global_id 14979 payload 274 2024-09-13T11:57:21.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d2b59e640 1 -- 172.21.15.100:0/3844058001 >> v1:172.21.15.122:6789/0 conn(0x7f3d2c0a2860 legacy=0x7f3d2c06f540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:57:21.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d2b59e640 1 -- 172.21.15.100:0/3844058001 >> v1:172.21.15.100:6789/0 conn(0x7f3d2c06ea50 legacy=0x7f3d2c06ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:21.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d2b59e640 10 monclient: _finish_hunting 0 2024-09-13T11:57:21.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d2b59e640 1 monclient: found mon.noname-a 2024-09-13T11:57:21.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d2b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:21.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d2b59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:57:21.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d2b59e640 1 -- 172.21.15.100:0/3844058001 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d2c0a1b50 con 0x7f3d2c13da00 2024-09-13T11:57:21.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d29d9b640 1 -- 172.21.15.100:0/3844058001 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 866934888 0 0) 0x7f3d2c0a1590 con 0x7f3d2c06ea50 2024-09-13T11:57:21.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.601+0000 7f3d29d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:57:21.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.602+0000 7f3d2b59e640 1 --2- 172.21.15.100:0/3844058001 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d2c13da00 0x7f3d2c13fdf0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009d90 tx=0x7f3d1c02a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6f2e75e0e9eb39c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:21.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.602+0000 7f3d29d9b640 1 -- 172.21.15.100:0/3844058001 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c007740 con 0x7f3d2c13da00 2024-09-13T11:57:21.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.602+0000 7f3d29d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:21.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.602+0000 7f3d29d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:57:21.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.602+0000 7f3d29d9b640 10 monclient: dump: 2024-09-13T11:57:21.614 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:21.614 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:21.614 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.614 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.614 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:21.615 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:21.615 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:21.615 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:21.615 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:21.615 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:21.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.602+0000 7f3d29d9b640 10 monclient: _finish_auth 0 2024-09-13T11:57:21.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.602+0000 7f3d29d9b640 10 monclient: _check_auth_tickets 2024-09-13T11:57:21.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.602+0000 7f3d29d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:21.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.602+0000 7f3d311ca640 5 monclient: authenticate success, global_id 14979 2024-09-13T11:57:21.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.602+0000 7f3d311ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:57:21.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.602+0000 7f3d29d9b640 1 -- 172.21.15.100:0/3844058001 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d1c0078e0 con 0x7f3d2c13da00 2024-09-13T11:57:21.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.602+0000 7f3d29d9b640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:21.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.602+0000 7f3d311ca640 10 monclient: get_monmap_and_config success 2024-09-13T11:57:21.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.602+0000 7f3d311ca640 4 set_mon_vals no callback set 2024-09-13T11:57:21.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.602+0000 7f3d29d9b640 1 -- 172.21.15.100:0/3844058001 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c007cb0 con 0x7f3d2c13da00 2024-09-13T11:57:21.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.602+0000 7f3d29d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:21.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.602+0000 7f3d29d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:21.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.602+0000 7f3d29d9b640 10 monclient: dump: 2024-09-13T11:57:21.616 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:21.616 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:21.616 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.616 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.616 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:21.616 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:21.616 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:21.617 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:21.617 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:21.617 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:21.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.602+0000 7f3d311ca640 10 monclient: shutdown 2024-09-13T11:57:21.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.602+0000 7f3d311ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:21.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.602+0000 7f3d311ca640 1 -- 172.21.15.100:0/3844058001 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d2c13da00 msgr2=0x7f3d2c13fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:21.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.602+0000 7f3d311ca640 1 --2- 172.21.15.100:0/3844058001 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d2c13da00 0x7f3d2c13fdf0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009d90 tx=0x7f3d1c02a530 comp rx=0 tx=0).stop 2024-09-13T11:57:21.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.603+0000 7f3d311ca640 1 -- 172.21.15.100:0/3844058001 shutdown_connections 2024-09-13T11:57:21.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.603+0000 7f3d311ca640 1 --2- 172.21.15.100:0/3844058001 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d2c13da00 0x7f3d2c13fdf0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:21.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.603+0000 7f3d311ca640 1 -- 172.21.15.100:0/3844058001 >> 172.21.15.100:0/3844058001 conn(0x7f3d2c0a68e0 msgr2=0x7f3d2c0a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:21.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.603+0000 7f3d311ca640 1 -- 172.21.15.100:0/3844058001 shutdown_connections 2024-09-13T11:57:21.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.603+0000 7f3d311ca640 1 -- 172.21.15.100:0/3844058001 wait complete. 2024-09-13T11:57:21.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.603+0000 7f3d311ca640 10 monclient: build_initial_monmap 2024-09-13T11:57:21.618 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.603+0000 7f3d311ca640 10 monclient: monmap: 2024-09-13T11:57:21.618 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:21.618 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:57:21.618 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:57:21.618 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:57:21.618 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:21.618 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:21.618 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:57:21.618 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:57:21.618 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:57:21.618 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:21.618 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d311ca640 1 Processor -- start 2024-09-13T11:57:21.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d311ca640 1 -- start start 2024-09-13T11:57:21.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d311ca640 10 monclient: init 2024-09-13T11:57:21.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d311ca640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:21.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d311ca640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:57:21.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d311ca640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d2c06ea50 0x7f3d2c09fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:21.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d311ca640 10 monclient(hunting): picked mon.noname-a con 0x7f3d2c06ea50 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:57:21.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d311ca640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d2c0a2860 0x7f3d2c1ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:21.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d311ca640 10 monclient(hunting): picked mon.noname-c con 0x7f3d2c0a2860 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:21.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d311ca640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d2c13da00 0x7f3d2c1cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:21.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d311ca640 10 monclient(hunting): picked mon.noname-b con 0x7f3d2c13da00 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:57:21.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d311ca640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:21.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d311ca640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c1404b0 con 0x7f3d2c06ea50 2024-09-13T11:57:21.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d311ca640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:21.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d311ca640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c140330 con 0x7f3d2c13da00 2024-09-13T11:57:21.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d311ca640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:21.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d311ca640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f3d2c140630 con 0x7f3d2c0a2860 2024-09-13T11:57:21.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d311ca640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:21.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d311ca640 10 monclient(hunting): authenticate will time out at 1022.197829s 2024-09-13T11:57:21.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d2ad9d640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d2c06ea50 0x7f3d2c09fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:21.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d2b59e640 1 -- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d2c13da00 msgr2=0x7f3d2c1cd0e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.122:3300/0 2024-09-13T11:57:21.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d2a59c640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d2c0a2860 0x7f3d2c1ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:21.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d2b59e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d2c13da00 0x7f3d2c1cd0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:57:21.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d2a59c640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d2c0a2860 0x7f3d2c1ccba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:54920/0 (socket says 172.21.15.100:54920) 2024-09-13T11:57:21.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d2ad9d640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d2c06ea50 0x7f3d2c09fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55942/0 (socket says 172.21.15.100:55942) 2024-09-13T11:57:21.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d2a59c640 1 -- 172.21.15.100:0/1725558635 learned_addr learned my addr 172.21.15.100:0/1725558635 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:21.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d2a59c640 10 monclient(hunting): get_auth_request con 0x7f3d2c0a2860 auth_method 0 2024-09-13T11:57:21.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d2a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:21.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d2a59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:21.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d2a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:21.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d2ad9d640 10 monclient(hunting): get_auth_request con 0x7f3d2c06ea50 auth_method 0 2024-09-13T11:57:21.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d2ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:21.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d2ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:21.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.604+0000 7f3d2ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:21.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d2a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:21.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d2a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:21.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d2a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:21.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d2ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:21.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d2ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:21.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d2ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:21.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d2a59c640 10 monclient(hunting): handle_auth_done global_id 24194 payload 824 2024-09-13T11:57:21.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d2a59c640 1 -- 172.21.15.100:0/1725558635 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d2c13da00 msgr2=0x7f3d2c1cd0e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:57:21.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d2a59c640 1 --2- 172.21.15.100:0/1725558635 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d2c13da00 0x7f3d2c1cd0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:21.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d2a59c640 1 -- 172.21.15.100:0/1725558635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d2c06ea50 msgr2=0x7f3d2c09fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:21.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d2a59c640 1 --2- 172.21.15.100:0/1725558635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d2c06ea50 0x7f3d2c09fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:21.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d2a59c640 10 monclient: _finish_hunting 0 2024-09-13T11:57:21.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d2a59c640 1 monclient: found mon.noname-c 2024-09-13T11:57:21.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d2a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:21.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d2a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:57:21.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d2a59c640 1 -- 172.21.15.100:0/1725558635 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d2c0a10d0 con 0x7f3d2c0a2860 2024-09-13T11:57:21.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d2ad9d640 1 --2- 172.21.15.100:0/1725558635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d2c06ea50 0x7f3d2c09fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:57:21.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d2a59c640 1 --2- 172.21.15.100:0/1725558635 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d2c0a2860 0x7f3d2c1ccba0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3d1400b580 tx=0x7f3d1400ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:21.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d237fe640 1 -- 172.21.15.100:0/1725558635 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d14016070 con 0x7f3d2c0a2860 2024-09-13T11:57:21.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:21.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d237fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:57:21.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d237fe640 10 monclient: dump: 2024-09-13T11:57:21.623 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:21.623 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:21.623 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.623 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.623 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:21.623 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:21.623 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:21.623 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:21.623 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:21.623 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:21.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d237fe640 10 monclient: _finish_auth 0 2024-09-13T11:57:21.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d237fe640 10 monclient: _check_auth_tickets 2024-09-13T11:57:21.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:21.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d237fe640 1 -- 172.21.15.100:0/1725558635 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d14003d90 con 0x7f3d2c0a2860 2024-09-13T11:57:21.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d311ca640 5 monclient: authenticate success, global_id 24194 2024-09-13T11:57:21.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d237fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:21.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d311ca640 10 monclient: _renew_subs 2024-09-13T11:57:21.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d237fe640 1 -- 172.21.15.100:0/1725558635 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d14005310 con 0x7f3d2c0a2860 2024-09-13T11:57:21.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d311ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:57:21.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d311ca640 1 -- 172.21.15.100:0/1725558635 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d2c120b60 con 0x7f3d2c0a2860 2024-09-13T11:57:21.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:21.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:57:21.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d2959a640 4 set_mon_vals no callback set 2024-09-13T11:57:21.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d237fe640 10 monclient: dump: 2024-09-13T11:57:21.625 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:21.625 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:21.625 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.625 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.625 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:21.625 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:21.625 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:21.625 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:21.625 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:21.625 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:21.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d311ca640 10 monclient: _renew_subs 2024-09-13T11:57:21.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d311ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:57:21.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.605+0000 7f3d311ca640 1 -- 172.21.15.100:0/1725558635 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d2c0a8ac0 con 0x7f3d2c0a2860 2024-09-13T11:57:21.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.606+0000 7f3d237fe640 1 -- 172.21.15.100:0/1725558635 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f3d14019070 con 0x7f3d2c0a2860 2024-09-13T11:57:21.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.606+0000 7f3d311ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:21.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.606+0000 7f3d311ca640 1 --2- 172.21.15.100:0/1725558635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d2c09d8e0 0x7f3d2c09dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:21.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.606+0000 7f3d311ca640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:21.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.606+0000 7f3d311ca640 1 -- 172.21.15.100:0/1725558635 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c06ee30 con 0x7f3d2c09d8e0 2024-09-13T11:57:21.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.606+0000 7f3d237fe640 1 --2- 172.21.15.100:0/1725558635 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f3d0c03df50 0x7f3d0c040410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:21.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.606+0000 7f3d2ad9d640 1 --2- 172.21.15.100:0/1725558635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d2c09d8e0 0x7f3d2c09dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:21.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.606+0000 7f3d2ad9d640 10 monclient: get_auth_request con 0x7f3d2c09d8e0 auth_method 0 2024-09-13T11:57:21.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.606+0000 7f3d2ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:21.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.606+0000 7f3d2ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:21.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.606+0000 7f3d2ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:21.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.606+0000 7f3d237fe640 1 -- 172.21.15.100:0/1725558635 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(226..226 src has 1..226) ==== 4346+0+0 (secure 0 0 0) 0x7f3d14051940 con 0x7f3d2c0a2860 2024-09-13T11:57:21.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.606+0000 7f3d2ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:21.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.606+0000 7f3d2ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:21.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.606+0000 7f3d2ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:21.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.606+0000 7f3d2ad9d640 10 monclient(hunting): handle_auth_done global_id 14988 payload 824 2024-09-13T11:57:21.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.606+0000 7f3d2ad9d640 1 -- 172.21.15.100:0/1725558635 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d2c0a07f0 con 0x7f3d2c09d8e0 2024-09-13T11:57:21.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.606+0000 7f3d2b59e640 1 --2- 172.21.15.100:0/1725558635 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f3d0c03df50 0x7f3d0c040410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:21.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.607+0000 7f3d2b59e640 10 monclient: get_auth_request con 0x7f3d0c03df50 auth_method 0 2024-09-13T11:57:21.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.607+0000 7f3d2ad9d640 1 --2- 172.21.15.100:0/1725558635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d2c09d8e0 0x7f3d2c09dcc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3d24001220 tx=0x7f3d24001d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:21.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.607+0000 7f3d237fe640 1 -- 172.21.15.100:0/1725558635 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2400f990 con 0x7f3d2c09d8e0 2024-09-13T11:57:21.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.607+0000 7f3d237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:21.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.607+0000 7f3d237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:21.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.607+0000 7f3d237fe640 10 monclient: dump: 2024-09-13T11:57:21.627 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:21.627 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:21.628 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.628 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.628 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:21.628 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:21.628 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:21.628 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:21.628 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:21.628 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:21.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.607+0000 7f3d237fe640 1 -- 172.21.15.100:0/1725558635 <== mon.0 v2:172.21.15.100:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3d24018070 con 0x7f3d2c09d8e0 2024-09-13T11:57:21.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.607+0000 7f3d237fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:21.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.607+0000 7f3d237fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:57:21.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.607+0000 7f3d2b59e640 1 --2- 172.21.15.100:0/1725558635 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f3d0c03df50 0x7f3d0c040410 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c034000 tx=0x7f3d1c02f000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:21.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.607+0000 7f3d237fe640 1 -- 172.21.15.100:0/1725558635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d2c09d8e0 msgr2=0x7f3d2c09dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:21.629 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.607+0000 7f3d237fe640 1 --2- 172.21.15.100:0/1725558635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d2c09d8e0 0x7f3d2c09dcc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3d24001220 tx=0x7f3d24001d60 comp rx=0 tx=0).stop 2024-09-13T11:57:21.629 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.607+0000 7f3d237fe640 1 -- 172.21.15.100:0/1725558635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d2c09d8e0 msgr2=0x7f3d2c09dcc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:57:21.629 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.607+0000 7f3d237fe640 1 --2- 172.21.15.100:0/1725558635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d2c09d8e0 0x7f3d2c09dcc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:21.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.613+0000 7f3d311ca640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:57:21.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.613+0000 7f3d311ca640 1 --2- 172.21.15.100:0/1725558635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d2c09e200 0x7f3d2c13b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:21.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.613+0000 7f3d311ca640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:21.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.613+0000 7f3d311ca640 1 -- 172.21.15.100:0/1725558635 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c069100 con 0x7f3d2c09e200 2024-09-13T11:57:21.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.613+0000 7f3d2ad9d640 1 --2- 172.21.15.100:0/1725558635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d2c09e200 0x7f3d2c13b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:21.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.613+0000 7f3d2ad9d640 10 monclient: get_auth_request con 0x7f3d2c09e200 auth_method 0 2024-09-13T11:57:21.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.613+0000 7f3d2ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:21.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.613+0000 7f3d2ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:21.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.613+0000 7f3d2ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:21.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.613+0000 7f3d2ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:21.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.613+0000 7f3d2ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:21.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.613+0000 7f3d2ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:21.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.613+0000 7f3d2ad9d640 10 monclient(hunting): handle_auth_done global_id 14991 payload 824 2024-09-13T11:57:21.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.613+0000 7f3d2ad9d640 1 -- 172.21.15.100:0/1725558635 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3d2c137ad0 con 0x7f3d2c09e200 2024-09-13T11:57:21.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.613+0000 7f3d2ad9d640 1 --2- 172.21.15.100:0/1725558635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d2c09e200 0x7f3d2c13b9f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3d24001220 tx=0x7f3d24001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:21.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.614+0000 7f3d237fe640 1 -- 172.21.15.100:0/1725558635 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24013740 con 0x7f3d2c09e200 2024-09-13T11:57:21.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.614+0000 7f3d237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:21.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.614+0000 7f3d237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:21.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.614+0000 7f3d237fe640 10 monclient: dump: 2024-09-13T11:57:21.632 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:21.632 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:21.632 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.632 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.632 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:21.632 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:21.632 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:21.633 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:21.633 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:21.633 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:21.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.614+0000 7f3d237fe640 1 -- 172.21.15.100:0/1725558635 <== mon.0 v2:172.21.15.100:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7f3d24013740 con 0x7f3d2c09e200 2024-09-13T11:57:21.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.614+0000 7f3d237fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:57:21.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.614+0000 7f3d237fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:57:21.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.614+0000 7f3d237fe640 1 -- 172.21.15.100:0/1725558635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d2c09e200 msgr2=0x7f3d2c13b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:21.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.614+0000 7f3d237fe640 1 --2- 172.21.15.100:0/1725558635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d2c09e200 0x7f3d2c13b9f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3d24001220 tx=0x7f3d24001080 comp rx=0 tx=0).stop 2024-09-13T11:57:21.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.614+0000 7f3d237fe640 1 -- 172.21.15.100:0/1725558635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d2c09e200 msgr2=0x7f3d2c13b9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:57:21.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.614+0000 7f3d237fe640 1 --2- 172.21.15.100:0/1725558635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d2c09e200 0x7f3d2c13b9f0 secure :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3d24001220 tx=0x7f3d24001080 comp rx=0 tx=0).stop 2024-09-13T11:57:21.633 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:57:21.634 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:57:21.634 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:57:21.634 INFO:teuthology.orchestra.run.smithi100.stdout: "state": "leader", 2024-09-13T11:57:21.634 INFO:teuthology.orchestra.run.smithi100.stdout: "election_epoch": 26, 2024-09-13T11:57:21.634 INFO:teuthology.orchestra.run.smithi100.stdout: "uptime": 142614, 2024-09-13T11:57:21.634 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum": [ 2024-09-13T11:57:21.634 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-13T11:57:21.634 INFO:teuthology.orchestra.run.smithi100.stdout: 2 2024-09-13T11:57:21.634 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:21.634 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_age": 2, 2024-09-13T11:57:21.634 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:57:21.634 INFO:teuthology.orchestra.run.smithi100.stdout: "required_con": "2450521708007866372", 2024-09-13T11:57:21.635 INFO:teuthology.orchestra.run.smithi100.stdout: "required_mon": [ 2024-09-13T11:57:21.635 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:57:21.635 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:57:21.635 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:57:21.635 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:57:21.635 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:57:21.635 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:57:21.635 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:57:21.635 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:57:21.635 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:57:21.635 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:57:21.635 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:57:21.635 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:21.636 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:57:21.636 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_mon": [ 2024-09-13T11:57:21.636 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:57:21.636 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:57:21.636 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:57:21.636 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:57:21.636 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:57:21.636 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:57:21.636 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:57:21.636 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:57:21.636 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:57:21.636 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:57:21.636 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:57:21.636 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:21.637 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:21.637 INFO:teuthology.orchestra.run.smithi100.stdout: "outside_quorum": [], 2024-09-13T11:57:21.637 INFO:teuthology.orchestra.run.smithi100.stdout: "extra_probe_peers": [], 2024-09-13T11:57:21.637 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_provider": [], 2024-09-13T11:57:21.637 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": { 2024-09-13T11:57:21.637 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-09-13T11:57:21.637 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "0b1f4b4d-fad4-4dd3-86ac-a286d80846c0", 2024-09-13T11:57:21.637 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:57:21.637 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:57:21.637 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release": 19, 2024-09-13T11:57:21.637 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release_name": "squid", 2024-09-13T11:57:21.637 INFO:teuthology.orchestra.run.smithi100.stdout: "election_strategy": 1, 2024-09-13T11:57:21.637 INFO:teuthology.orchestra.run.smithi100.stdout: "disallowed_leaders: ": "", 2024-09-13T11:57:21.638 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false, 2024-09-13T11:57:21.638 INFO:teuthology.orchestra.run.smithi100.stdout: "tiebreaker_mon": "", 2024-09-13T11:57:21.638 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_ranks: ": "", 2024-09-13T11:57:21.638 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:57:21.638 INFO:teuthology.orchestra.run.smithi100.stdout: "persistent": [ 2024-09-13T11:57:21.638 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:57:21.638 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:57:21.638 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:57:21.638 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:57:21.638 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:57:21.638 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:57:21.638 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:57:21.638 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:57:21.638 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:57:21.639 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:57:21.639 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:57:21.639 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:21.639 INFO:teuthology.orchestra.run.smithi100.stdout: "optional": [] 2024-09-13T11:57:21.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:21.639 INFO:teuthology.orchestra.run.smithi100.stdout: "mons": [ 2024-09-13T11:57:21.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.639 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:57:21.639 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:57:21.639 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:57:21.639 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:57:21.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.640 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:57:21.640 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3300", 2024-09-13T11:57:21.640 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:21.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:21.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.640 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:57:21.640 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789", 2024-09-13T11:57:21.640 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:21.640 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:21.640 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:21.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:21.640 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789/0", 2024-09-13T11:57:21.640 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-13T11:57:21.640 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:57:21.641 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:57:21.641 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:57:21.641 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:21.641 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.641 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:57:21.641 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:57:21.641 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:57:21.641 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:57:21.641 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.641 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:57:21.641 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:3300", 2024-09-13T11:57:21.641 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:21.641 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:21.642 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.642 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:57:21.642 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789", 2024-09-13T11:57:21.642 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:21.642 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:21.642 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:21.642 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:21.642 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789/0", 2024-09-13T11:57:21.642 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.122:6789/0", 2024-09-13T11:57:21.642 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:57:21.642 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:57:21.642 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:57:21.642 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:21.642 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.643 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:57:21.643 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:57:21.643 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:57:21.643 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:57:21.643 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.643 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:57:21.643 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3301", 2024-09-13T11:57:21.643 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:21.643 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:21.643 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.643 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:57:21.643 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790", 2024-09-13T11:57:21.643 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:21.644 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:21.644 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:21.644 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:21.644 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790/0", 2024-09-13T11:57:21.644 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-13T11:57:21.644 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:57:21.644 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:57:21.644 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:57:21.644 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:21.644 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:21.644 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:21.644 INFO:teuthology.orchestra.run.smithi100.stdout: "feature_map": { 2024-09-13T11:57:21.644 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": [ 2024-09-13T11:57:21.645 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.645 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:21.645 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:21.645 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:57:21.645 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:21.645 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:21.645 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": [ 2024-09-13T11:57:21.645 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.645 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:21.645 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:21.645 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 3 2024-09-13T11:57:21.645 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:21.645 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:21.646 INFO:teuthology.orchestra.run.smithi100.stdout: "client": [ 2024-09-13T11:57:21.646 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.646 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:21.646 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:21.646 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 4 2024-09-13T11:57:21.646 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:21.646 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:21.646 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": [ 2024-09-13T11:57:21.646 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.646 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:21.646 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:21.646 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:57:21.646 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:21.646 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:21.647 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:21.647 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false 2024-09-13T11:57:21.647 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:57:21.648 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.614+0000 7f3d311ca640 1 -- 172.21.15.100:0/1725558635 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f3d0c03df50 msgr2=0x7f3d0c040410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:21.648 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.614+0000 7f3d311ca640 1 --2- 172.21.15.100:0/1725558635 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f3d0c03df50 0x7f3d0c040410 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c034000 tx=0x7f3d1c02f000 comp rx=0 tx=0).stop 2024-09-13T11:57:21.648 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.614+0000 7f3d311ca640 10 monclient: shutdown 2024-09-13T11:57:21.648 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.614+0000 7f3d311ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:21.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.614+0000 7f3d311ca640 1 -- 172.21.15.100:0/1725558635 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d2c0a2860 msgr2=0x7f3d2c1ccba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:21.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.614+0000 7f3d311ca640 1 --2- 172.21.15.100:0/1725558635 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d2c0a2860 0x7f3d2c1ccba0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3d1400b580 tx=0x7f3d1400ba40 comp rx=0 tx=0).stop 2024-09-13T11:57:21.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.614+0000 7f3d2b59e640 1 -- 172.21.15.100:0/1725558635 reap_dead start 2024-09-13T11:57:21.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.614+0000 7f3d311ca640 1 -- 172.21.15.100:0/1725558635 shutdown_connections 2024-09-13T11:57:21.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.614+0000 7f3d311ca640 1 --2- 172.21.15.100:0/1725558635 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d2c0a2860 0x7f3d2c1ccba0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:21.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.614+0000 7f3d311ca640 1 -- 172.21.15.100:0/1725558635 >> 172.21.15.100:0/1725558635 conn(0x7f3d2c0a68e0 msgr2=0x7f3d2c13dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:21.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.614+0000 7f3d311ca640 1 -- 172.21.15.100:0/1725558635 shutdown_connections 2024-09-13T11:57:21.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.614+0000 7f3d311ca640 1 -- 172.21.15.100:0/1725558635 wait complete. 2024-09-13T11:57:21.649 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-13T11:57:21.649 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-13T11:57:21.650 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T11:57:21.747 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.746+0000 7f6989a40640 10 monclient: get_monmap_and_config 2024-09-13T11:57:21.748 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.746+0000 7f6989a40640 10 monclient: build_initial_monmap 2024-09-13T11:57:21.748 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.746+0000 7f6989a40640 10 monclient: monmap: 2024-09-13T11:57:21.748 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:21.748 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:21.748 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:57:21.747481+0000 2024-09-13T11:57:21.748 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:57:21.747481+0000 2024-09-13T11:57:21.748 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:21.748 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:21.748 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:57:21.748 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:57:21.748 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:57:21.749 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:57:21.749 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:57:21.749 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:21.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.747+0000 7f6989a40640 1 Processor -- start 2024-09-13T11:57:21.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.747+0000 7f6989a40640 1 -- start start 2024-09-13T11:57:21.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.747+0000 7f6989a40640 10 monclient: init 2024-09-13T11:57:21.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.747+0000 7f6989a40640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:21.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.747+0000 7f6989a40640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-13T11:57:21.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.747+0000 7f6989a40640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6984137a60 0x7f69840a2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:21.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.747+0000 7f6989a40640 10 monclient(hunting): picked mon.noname-c con 0x7f6984137a60 addr v2:172.21.15.122:3300/0 2024-09-13T11:57:21.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.747+0000 7f6989a40640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6984137e40 addr v1:172.21.15.100:6789/0 2024-09-13T11:57:21.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.747+0000 7f6989a40640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6984138250 addr v1:172.21.15.122:6789/0 2024-09-13T11:57:21.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.747+0000 7f6989a40640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69840a5770 con 0x7f6984137e40 2024-09-13T11:57:21.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.747+0000 7f6989a40640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69840a5be0 con 0x7f6984138250 2024-09-13T11:57:21.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.747+0000 7f6989a40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:21.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.747+0000 7f6989a40640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f69840a4410 con 0x7f6984137a60 2024-09-13T11:57:21.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.747+0000 7f6989a40640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:21.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.747+0000 7f6989a40640 10 monclient(hunting): authenticate will time out at 1022.340855s 2024-09-13T11:57:21.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.747+0000 7f6982ffd640 1 -- >> v2:172.21.15.122:3300/0 conn(0x7f6984137a60 msgr2=0x7f69840a2980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.122:3300/0 2024-09-13T11:57:21.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.747+0000 7f69827fc640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f6984137e40 0x7f69840a4950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:37336/0 (socket says 172.21.15.100:37336) 2024-09-13T11:57:21.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.747+0000 7f69827fc640 1 -- 172.21.15.100:0/2126273869 learned_addr learned my addr 172.21.15.100:0/2126273869 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:57:21.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.747+0000 7f6982ffd640 1 --2- 172.21.15.100:0/2126273869 >> v2:172.21.15.122:3300/0 conn(0x7f6984137a60 0x7f69840a2980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:57:21.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.747+0000 7f69837fe640 1 -- 172.21.15.100:0/2126273869 >> v1:172.21.15.122:6789/0 conn(0x7f6984138250 legacy=0x7f69840a5060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6789/0 2024-09-13T11:57:21.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 1 -- 172.21.15.100:0/2126273869 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3306723665 0 0) 0x7f69840a5770 con 0x7f6984137e40 2024-09-13T11:57:21.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 10 monclient(hunting): _negotiate 2024-09-13T11:57:21.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:21.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:21.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 10 monclient(hunting): my global_id is 14994 2024-09-13T11:57:21.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 1 -- 172.21.15.100:0/2126273869 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f696c003520 con 0x7f6984137e40 2024-09-13T11:57:21.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 1 -- 172.21.15.100:0/2126273869 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 427481020 0 0) 0x7f696c003520 con 0x7f6984137e40 2024-09-13T11:57:21.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 1 -- 172.21.15.100:0/2126273869 >> v2:172.21.15.122:3300/0 conn(0x7f6984137a60 msgr2=0x7f69840a2980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:57:21.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 1 --2- 172.21.15.100:0/2126273869 >> v2:172.21.15.122:3300/0 conn(0x7f6984137a60 0x7f69840a2980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:21.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 1 -- 172.21.15.100:0/2126273869 >> v1:172.21.15.122:6789/0 conn(0x7f6984138250 legacy=0x7f69840a5060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:57:21.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 10 monclient: _finish_hunting 0 2024-09-13T11:57:21.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 1 monclient: found mon.noname-a-legacy 2024-09-13T11:57:21.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:21.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.100:6789/0 2024-09-13T11:57:21.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 1 -- 172.21.15.100:0/2126273869 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69840a76b0 con 0x7f6984137e40 2024-09-13T11:57:21.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 10 monclient: _finish_auth 0 2024-09-13T11:57:21.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:57:21.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:21.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 1 -- 172.21.15.100:0/2126273869 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 626700363 0 0) 0x7f69740016f0 con 0x7f6984137e40 2024-09-13T11:57:21.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:21.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-13T11:57:21.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 10 monclient: dump: 2024-09-13T11:57:21.763 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:21.763 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:21.763 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.763 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.763 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:21.763 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:21.763 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:21.763 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:21.764 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:21.764 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:21.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] but i'm connected to v1:172.21.15.100:6789/0, reconnecting 2024-09-13T11:57:21.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:21.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 1 -- 172.21.15.100:0/2126273869 >> v1:172.21.15.100:6789/0 conn(0x7f6984137e40 legacy=0x7f69840a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:21.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:57:21.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 1 --2- 172.21.15.100:0/2126273869 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f696c0039b0 0x7f696c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:21.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 10 monclient(hunting): picked mon.c con 0x7f696c0039b0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:21.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 1 --2- 172.21.15.100:0/2126273869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f696c004800 0x7f696c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:21.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 10 monclient(hunting): picked mon.b con 0x7f696c004800 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:57:21.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 1 --2- 172.21.15.100:0/2126273869 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f696c0053c0 0x7f696c00ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:21.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 10 monclient(hunting): picked mon.a con 0x7f696c0053c0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:57:21.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:21.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 1 -- 172.21.15.100:0/2126273869 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f696c00c2b0 con 0x7f696c0053c0 2024-09-13T11:57:21.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:21.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 1 -- 172.21.15.100:0/2126273869 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f696c00c460 con 0x7f696c004800 2024-09-13T11:57:21.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:21.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 1 -- 172.21.15.100:0/2126273869 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f696c00c660 con 0x7f696c0039b0 2024-09-13T11:57:21.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:21.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6981ffb640 10 monclient(hunting): _finish_auth 0 2024-09-13T11:57:21.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f69827fc640 1 -- 172.21.15.100:0/2126273869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f696c004800 msgr2=0x7f696c004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.122:3300/0 2024-09-13T11:57:21.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6982ffd640 1 --2- 172.21.15.100:0/2126273869 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f696c0039b0 0x7f696c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:21.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f69837fe640 1 --2- 172.21.15.100:0/2126273869 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f696c0053c0 0x7f696c00ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:21.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6982ffd640 10 monclient(hunting): get_auth_request con 0x7f696c0039b0 auth_method 0 2024-09-13T11:57:21.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6982ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:21.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6982ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:21.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6982ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:57:21.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f69837fe640 10 monclient(hunting): get_auth_request con 0x7f696c0053c0 auth_method 0 2024-09-13T11:57:21.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f69837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:21.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f69837fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:21.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f69837fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:57:21.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f69827fc640 1 --2- 172.21.15.100:0/2126273869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f696c004800 0x7f696c004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:57:21.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6982ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:21.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6982ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:21.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6982ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:57:21.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f69837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:21.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f69837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:21.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f69837fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:57:21.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f69837fe640 10 monclient(hunting): handle_auth_done global_id 14994 payload 293 2024-09-13T11:57:21.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f69837fe640 1 -- 172.21.15.100:0/2126273869 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f696c0039b0 msgr2=0x7f696c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:57:21.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f69837fe640 1 --2- 172.21.15.100:0/2126273869 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f696c0039b0 0x7f696c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:21.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f69837fe640 1 -- 172.21.15.100:0/2126273869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f696c004800 msgr2=0x7f696c004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:57:21.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f69837fe640 1 --2- 172.21.15.100:0/2126273869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f696c004800 0x7f696c004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:21.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f69837fe640 10 monclient: _finish_hunting 0 2024-09-13T11:57:21.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f69837fe640 1 monclient: found mon.a 2024-09-13T11:57:21.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f69837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:21.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f69837fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:21.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f69837fe640 1 -- 172.21.15.100:0/2126273869 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f696c003520 con 0x7f696c0053c0 2024-09-13T11:57:21.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f69837fe640 10 monclient: _finish_auth 0 2024-09-13T11:57:21.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f69837fe640 10 monclient: _check_auth_tickets 2024-09-13T11:57:21.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f69837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:21.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6989a40640 5 monclient: authenticate success, global_id 14994 2024-09-13T11:57:21.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6989a40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:57:21.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.748+0000 7f6982ffd640 1 --2- 172.21.15.100:0/2126273869 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f696c0039b0 0x7f696c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:57:21.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.749+0000 7f69837fe640 1 -- 172.21.15.100:0/2126273869 reap_dead start 2024-09-13T11:57:21.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.749+0000 7f69837fe640 1 --2- 172.21.15.100:0/2126273869 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f696c0053c0 0x7f696c00ba00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f697c009e30 tx=0x7f696c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f885aa85bce9adcb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:21.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.749+0000 7f6981ffb640 1 -- 172.21.15.100:0/2126273869 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6978009500 con 0x7f696c0053c0 2024-09-13T11:57:21.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.749+0000 7f6981ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:21.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.749+0000 7f6981ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:21.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.749+0000 7f6981ffb640 10 monclient: dump: 2024-09-13T11:57:21.768 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:21.768 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:21.768 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.768 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.768 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:21.768 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:21.769 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:21.769 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:21.769 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:21.769 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:21.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.749+0000 7f6989a40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:57:21.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.749+0000 7f6981ffb640 1 -- 172.21.15.100:0/2126273869 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f697c02c040 con 0x7f696c0053c0 2024-09-13T11:57:21.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.749+0000 7f6981ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:21.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.749+0000 7f6981ffb640 1 -- 172.21.15.100:0/2126273869 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c009a30 con 0x7f696c0053c0 2024-09-13T11:57:21.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.749+0000 7f6989a40640 10 monclient: get_monmap_and_config success 2024-09-13T11:57:21.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.749+0000 7f6989a40640 4 set_mon_vals no callback set 2024-09-13T11:57:21.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.749+0000 7f6981ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:21.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.749+0000 7f6981ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:21.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.749+0000 7f6981ffb640 10 monclient: dump: 2024-09-13T11:57:21.770 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:21.770 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:21.770 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.770 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.770 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:21.770 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:21.770 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:21.770 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:21.770 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:21.770 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:21.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.750+0000 7f6989a40640 10 monclient: shutdown 2024-09-13T11:57:21.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.750+0000 7f6989a40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:21.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.750+0000 7f6989a40640 1 -- 172.21.15.100:0/2126273869 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f696c0053c0 msgr2=0x7f696c00ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:21.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.750+0000 7f6989a40640 1 --2- 172.21.15.100:0/2126273869 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f696c0053c0 0x7f696c00ba00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f697c009e30 tx=0x7f696c005830 comp rx=0 tx=0).stop 2024-09-13T11:57:21.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.750+0000 7f6989a40640 1 -- 172.21.15.100:0/2126273869 shutdown_connections 2024-09-13T11:57:21.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.750+0000 7f6989a40640 1 --2- 172.21.15.100:0/2126273869 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f696c0053c0 0x7f696c00ba00 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:21.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.750+0000 7f6989a40640 1 -- 172.21.15.100:0/2126273869 >> 172.21.15.100:0/2126273869 conn(0x7f69840aa580 msgr2=0x7f69840a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:21.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.750+0000 7f6989a40640 1 -- 172.21.15.100:0/2126273869 shutdown_connections 2024-09-13T11:57:21.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.750+0000 7f6989a40640 1 -- 172.21.15.100:0/2126273869 wait complete. 2024-09-13T11:57:21.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.750+0000 7f6989a40640 10 monclient: build_initial_monmap 2024-09-13T11:57:21.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.750+0000 7f6989a40640 10 monclient: monmap: 2024-09-13T11:57:21.771 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:57:21.771 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:57:21.772 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:57:21.772 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:57:21.772 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:57:21.772 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:21.772 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:57:21.772 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:57:21.772 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:57:21.772 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:21.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f6989a40640 1 Processor -- start 2024-09-13T11:57:21.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f6989a40640 1 -- start start 2024-09-13T11:57:21.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f6989a40640 10 monclient: init 2024-09-13T11:57:21.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f6989a40640 10 monclient: _reopen_session rank -1 2024-09-13T11:57:21.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f6989a40640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:57:21.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f6989a40640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f696c0053c0 0x7f69841cc770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:21.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f6989a40640 10 monclient(hunting): picked mon.noname-c con 0x7f696c0053c0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:57:21.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f6989a40640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69841cccb0 0x7f69841240a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:21.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f6989a40640 10 monclient(hunting): picked mon.noname-a con 0x7f69841cccb0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:57:21.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f6989a40640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69841cd110 0x7f69841245e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:21.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f6989a40640 10 monclient(hunting): picked mon.noname-b con 0x7f69841cd110 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:57:21.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f6989a40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:21.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f6989a40640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f698406fd80 con 0x7f69841cccb0 2024-09-13T11:57:21.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f6989a40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:21.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f6989a40640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f69840a1700 con 0x7f69841cd110 2024-09-13T11:57:21.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f6989a40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:21.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f6989a40640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f698406ff00 con 0x7f696c0053c0 2024-09-13T11:57:21.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f6989a40640 10 monclient(hunting): _renew_subs 2024-09-13T11:57:21.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f6989a40640 10 monclient(hunting): authenticate will time out at 1022.344858s 2024-09-13T11:57:21.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f6982ffd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f696c0053c0 0x7f69841cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:21.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f69827fc640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69841cccb0 0x7f69841240a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:21.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f69837fe640 1 -- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69841cd110 msgr2=0x7f69841245e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.122:3300/0 2024-09-13T11:57:21.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f69837fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69841cd110 0x7f69841245e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:57:21.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f69827fc640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69841cccb0 0x7f69841240a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55986/0 (socket says 172.21.15.100:55986) 2024-09-13T11:57:21.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f6982ffd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f696c0053c0 0x7f69841cc770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:54942/0 (socket says 172.21.15.100:54942) 2024-09-13T11:57:21.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.751+0000 7f69827fc640 1 -- 172.21.15.100:0/1178896228 learned_addr learned my addr 172.21.15.100:0/1178896228 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:57:21.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f69827fc640 10 monclient(hunting): get_auth_request con 0x7f69841cccb0 auth_method 0 2024-09-13T11:57:21.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f69827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:21.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f69827fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:21.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f69827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:21.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f6982ffd640 10 monclient(hunting): get_auth_request con 0x7f696c0053c0 auth_method 0 2024-09-13T11:57:21.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f6982ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:21.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f6982ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:21.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f6982ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:21.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f69827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:21.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f69827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:21.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f69827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:21.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f6982ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:21.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f6982ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:21.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f6982ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:21.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f69827fc640 10 monclient(hunting): handle_auth_done global_id 14997 payload 824 2024-09-13T11:57:21.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f69827fc640 1 -- 172.21.15.100:0/1178896228 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f696c0053c0 msgr2=0x7f69841cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:21.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f69827fc640 1 --2- 172.21.15.100:0/1178896228 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f696c0053c0 0x7f69841cc770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:21.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f69827fc640 1 -- 172.21.15.100:0/1178896228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69841cd110 msgr2=0x7f69841245e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:57:21.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f69827fc640 1 --2- 172.21.15.100:0/1178896228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69841cd110 0x7f69841245e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:57:21.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f69827fc640 10 monclient: _finish_hunting 0 2024-09-13T11:57:21.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f69827fc640 1 monclient: found mon.noname-a 2024-09-13T11:57:21.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f69827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:57:21.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f69827fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:57:21.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f69827fc640 1 -- 172.21.15.100:0/1178896228 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69840a88d0 con 0x7f69841cccb0 2024-09-13T11:57:21.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f69827fc640 1 --2- 172.21.15.100:0/1178896228 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69841cccb0 0x7f69841240a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f697400c730 tx=0x7f697400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:21.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f696bfff640 1 -- 172.21.15.100:0/1178896228 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974018070 con 0x7f69841cccb0 2024-09-13T11:57:21.777 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f696bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:21.777 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f696bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:57:21.777 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f696bfff640 10 monclient: dump: 2024-09-13T11:57:21.777 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:21.777 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:21.777 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.777 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.777 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:21.777 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:21.777 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:21.777 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:21.777 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:21.778 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:21.778 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f696bfff640 10 monclient: _finish_auth 0 2024-09-13T11:57:21.778 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f696bfff640 10 monclient: _check_auth_tickets 2024-09-13T11:57:21.778 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f696bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:57:21.778 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f696bfff640 1 -- 172.21.15.100:0/1178896228 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6974007500 con 0x7f69841cccb0 2024-09-13T11:57:21.778 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f6989a40640 5 monclient: authenticate success, global_id 14997 2024-09-13T11:57:21.778 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f6989a40640 10 monclient: _renew_subs 2024-09-13T11:57:21.778 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f6989a40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:21.778 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f6989a40640 1 -- 172.21.15.100:0/1178896228 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69840a1140 con 0x7f69841cccb0 2024-09-13T11:57:21.778 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f696bfff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:57:21.778 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f696bfff640 1 -- 172.21.15.100:0/1178896228 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974004a20 con 0x7f69841cccb0 2024-09-13T11:57:21.778 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f696bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:21.778 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f696bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:57:21.778 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f696bfff640 10 monclient: dump: 2024-09-13T11:57:21.779 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:21.779 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:21.779 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.779 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.779 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:21.779 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:21.779 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:21.779 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:21.779 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:21.779 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:21.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f6989a40640 10 monclient: _renew_subs 2024-09-13T11:57:21.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f6989a40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:57:21.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f6989a40640 1 -- 172.21.15.100:0/1178896228 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6984127ec0 con 0x7f69841cccb0 2024-09-13T11:57:21.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.752+0000 7f69817fa640 4 set_mon_vals no callback set 2024-09-13T11:57:21.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.753+0000 7f6989a40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:21.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.753+0000 7f6989a40640 1 --2- 172.21.15.100:0/1178896228 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6984124b20 0x7f6984124f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:21.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.753+0000 7f6989a40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:21.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.753+0000 7f6989a40640 1 -- 172.21.15.100:0/1178896228 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f69840a2b30 con 0x7f6984124b20 2024-09-13T11:57:21.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.753+0000 7f696bfff640 1 -- 172.21.15.100:0/1178896228 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f697401b070 con 0x7f69841cccb0 2024-09-13T11:57:21.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.753+0000 7f6982ffd640 1 --2- 172.21.15.100:0/1178896228 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6984124b20 0x7f6984124f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:21.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.753+0000 7f6982ffd640 10 monclient: get_auth_request con 0x7f6984124b20 auth_method 0 2024-09-13T11:57:21.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.753+0000 7f6982ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:21.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.753+0000 7f6982ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:21.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.753+0000 7f6982ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:21.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.753+0000 7f6982ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:21.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.753+0000 7f6982ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:21.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.753+0000 7f6982ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:21.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.753+0000 7f696bfff640 1 --2- 172.21.15.100:0/1178896228 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f697003df50 0x7f6970040410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:21.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.753+0000 7f6982ffd640 10 monclient(hunting): handle_auth_done global_id 24200 payload 824 2024-09-13T11:57:21.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.753+0000 7f696bfff640 1 -- 172.21.15.100:0/1178896228 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 4346+0+0 (secure 0 0 0) 0x7f6974051980 con 0x7f69841cccb0 2024-09-13T11:57:21.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.753+0000 7f6982ffd640 1 -- 172.21.15.100:0/1178896228 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6984126720 con 0x7f6984124b20 2024-09-13T11:57:21.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.753+0000 7f6982ffd640 1 --2- 172.21.15.100:0/1178896228 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6984124b20 0x7f6984124f00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6978011860 tx=0x7f6978011d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:21.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.753+0000 7f696bfff640 1 -- 172.21.15.100:0/1178896228 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6978018070 con 0x7f6984124b20 2024-09-13T11:57:21.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.753+0000 7f696bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:21.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.753+0000 7f696bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:57:21.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.753+0000 7f696bfff640 10 monclient: dump: 2024-09-13T11:57:21.781 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:21.781 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:21.781 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.781 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.782 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:21.782 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:21.782 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:21.782 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:21.782 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:21.782 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:21.782 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.753+0000 7f69837fe640 1 --2- 172.21.15.100:0/1178896228 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f697003df50 0x7f6970040410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:21.782 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.754+0000 7f69837fe640 10 monclient: get_auth_request con 0x7f697003df50 auth_method 0 2024-09-13T11:57:21.782 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.754+0000 7f696bfff640 1 -- 172.21.15.100:0/1178896228 <== mon.2 v2:172.21.15.100:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6978003bc0 con 0x7f6984124b20 2024-09-13T11:57:21.782 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.754+0000 7f696bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:57:21.782 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.754+0000 7f696bfff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:57:21.782 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.754+0000 7f696bfff640 1 -- 172.21.15.100:0/1178896228 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6984124b20 msgr2=0x7f6984124f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:21.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.754+0000 7f696bfff640 1 --2- 172.21.15.100:0/1178896228 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6984124b20 0x7f6984124f00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6978011860 tx=0x7f6978011d20 comp rx=0 tx=0).stop 2024-09-13T11:57:21.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.754+0000 7f696bfff640 1 -- 172.21.15.100:0/1178896228 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6984124b20 msgr2=0x7f6984124f00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:57:21.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.754+0000 7f696bfff640 1 --2- 172.21.15.100:0/1178896228 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6984124b20 0x7f6984124f00 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6978011860 tx=0x7f6978011d20 comp rx=0 tx=0).stop 2024-09-13T11:57:21.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.754+0000 7f69837fe640 1 --2- 172.21.15.100:0/1178896228 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f697003df50 0x7f6970040410 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f696c00c430 tx=0x7f697c005000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:21.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.760+0000 7f6989a40640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:57:21.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.760+0000 7f6989a40640 1 --2- 172.21.15.100:0/1178896228 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69841205a0 0x7f69841209b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:57:21.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.760+0000 7f6989a40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:57:21.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.760+0000 7f6989a40640 1 -- 172.21.15.100:0/1178896228 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f6984125b60 con 0x7f69841205a0 2024-09-13T11:57:21.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.760+0000 7f6982ffd640 1 --2- 172.21.15.100:0/1178896228 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69841205a0 0x7f69841209b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:57:21.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.760+0000 7f6982ffd640 10 monclient: get_auth_request con 0x7f69841205a0 auth_method 0 2024-09-13T11:57:21.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.760+0000 7f6982ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:57:21.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.760+0000 7f6982ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:57:21.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.760+0000 7f6982ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:57:21.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.760+0000 7f6982ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:57:21.786 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.760+0000 7f6982ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:57:21.786 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.760+0000 7f6982ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:57:21.786 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.760+0000 7f6982ffd640 10 monclient(hunting): handle_auth_done global_id 24203 payload 824 2024-09-13T11:57:21.786 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.760+0000 7f6982ffd640 1 -- 172.21.15.100:0/1178896228 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6984125440 con 0x7f69841205a0 2024-09-13T11:57:21.786 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.760+0000 7f6982ffd640 1 --2- 172.21.15.100:0/1178896228 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69841205a0 0x7f69841209b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6978011860 tx=0x7f6978005a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:57:21.786 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.761+0000 7f696bfff640 1 -- 172.21.15.100:0/1178896228 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697800fe90 con 0x7f69841205a0 2024-09-13T11:57:21.786 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.761+0000 7f696bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:57:21.786 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.761+0000 7f696bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:57:21.786 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.761+0000 7f696bfff640 10 monclient: dump: 2024-09-13T11:57:21.786 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:57:21.786 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:57:21.786 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.786 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:57:21.787 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:57:21.787 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:57:21.787 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:57:21.787 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:57:21.787 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:57:21.787 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:57:21.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.761+0000 7f696bfff640 1 -- 172.21.15.100:0/1178896228 <== mon.2 v2:172.21.15.100:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7f697801d450 con 0x7f69841205a0 2024-09-13T11:57:21.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.761+0000 7f696bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:57:21.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.761+0000 7f696bfff640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:57:21.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.761+0000 7f696bfff640 1 -- 172.21.15.100:0/1178896228 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69841205a0 msgr2=0x7f69841209b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:21.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.761+0000 7f696bfff640 1 --2- 172.21.15.100:0/1178896228 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69841205a0 0x7f69841209b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6978011860 tx=0x7f6978005a10 comp rx=0 tx=0).stop 2024-09-13T11:57:21.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.761+0000 7f696bfff640 1 -- 172.21.15.100:0/1178896228 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69841205a0 msgr2=0x7f69841209b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:57:21.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.761+0000 7f696bfff640 1 --2- 172.21.15.100:0/1178896228 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69841205a0 0x7f69841209b0 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6978011860 tx=0x7f6978005a10 comp rx=0 tx=0).stop 2024-09-13T11:57:21.788 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:57:21.788 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:57:21.788 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:57:21.788 INFO:teuthology.orchestra.run.smithi100.stdout: "state": "peon", 2024-09-13T11:57:21.788 INFO:teuthology.orchestra.run.smithi100.stdout: "election_epoch": 26, 2024-09-13T11:57:21.788 INFO:teuthology.orchestra.run.smithi100.stdout: "uptime": 34739, 2024-09-13T11:57:21.788 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum": [ 2024-09-13T11:57:21.788 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-13T11:57:21.788 INFO:teuthology.orchestra.run.smithi100.stdout: 2 2024-09-13T11:57:21.789 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:21.789 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_age": 2, 2024-09-13T11:57:21.789 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:57:21.789 INFO:teuthology.orchestra.run.smithi100.stdout: "required_con": "2450521708007866372", 2024-09-13T11:57:21.789 INFO:teuthology.orchestra.run.smithi100.stdout: "required_mon": [ 2024-09-13T11:57:21.789 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:57:21.789 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:57:21.789 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:57:21.789 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:57:21.789 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:57:21.789 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:57:21.789 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:57:21.789 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:57:21.790 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:57:21.790 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:57:21.790 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:57:21.790 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:21.790 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:57:21.790 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_mon": [ 2024-09-13T11:57:21.790 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:57:21.790 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:57:21.790 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:57:21.790 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:57:21.790 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:57:21.790 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:57:21.790 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:57:21.791 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:57:21.791 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:57:21.791 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:57:21.791 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:57:21.791 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:21.791 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:21.791 INFO:teuthology.orchestra.run.smithi100.stdout: "outside_quorum": [], 2024-09-13T11:57:21.791 INFO:teuthology.orchestra.run.smithi100.stdout: "extra_probe_peers": [], 2024-09-13T11:57:21.791 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_provider": [], 2024-09-13T11:57:21.791 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": { 2024-09-13T11:57:21.791 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-09-13T11:57:21.791 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "0b1f4b4d-fad4-4dd3-86ac-a286d80846c0", 2024-09-13T11:57:21.791 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:57:21.792 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:57:21.792 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release": 19, 2024-09-13T11:57:21.792 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release_name": "squid", 2024-09-13T11:57:21.792 INFO:teuthology.orchestra.run.smithi100.stdout: "election_strategy": 1, 2024-09-13T11:57:21.792 INFO:teuthology.orchestra.run.smithi100.stdout: "disallowed_leaders: ": "", 2024-09-13T11:57:21.792 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false, 2024-09-13T11:57:21.792 INFO:teuthology.orchestra.run.smithi100.stdout: "tiebreaker_mon": "", 2024-09-13T11:57:21.792 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_ranks: ": "", 2024-09-13T11:57:21.792 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:57:21.792 INFO:teuthology.orchestra.run.smithi100.stdout: "persistent": [ 2024-09-13T11:57:21.792 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:57:21.793 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:57:21.793 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:57:21.793 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:57:21.793 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:57:21.793 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:57:21.793 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:57:21.793 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:57:21.793 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:57:21.793 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:57:21.793 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:57:21.793 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:21.793 INFO:teuthology.orchestra.run.smithi100.stdout: "optional": [] 2024-09-13T11:57:21.794 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:21.794 INFO:teuthology.orchestra.run.smithi100.stdout: "mons": [ 2024-09-13T11:57:21.794 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.794 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:57:21.794 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:57:21.794 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:57:21.794 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:57:21.794 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.794 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:57:21.794 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3300", 2024-09-13T11:57:21.794 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:21.795 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:21.795 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.795 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:57:21.795 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789", 2024-09-13T11:57:21.795 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:21.795 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:21.795 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:21.795 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:21.795 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789/0", 2024-09-13T11:57:21.795 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-13T11:57:21.795 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:57:21.795 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:57:21.795 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:57:21.796 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:21.796 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.796 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:57:21.796 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:57:21.796 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:57:21.796 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:57:21.796 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.796 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:57:21.796 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:3300", 2024-09-13T11:57:21.796 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:21.796 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:21.796 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.796 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:57:21.797 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789", 2024-09-13T11:57:21.797 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:21.797 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:21.797 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:21.797 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:21.797 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789/0", 2024-09-13T11:57:21.797 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.122:6789/0", 2024-09-13T11:57:21.797 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:57:21.797 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:57:21.797 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:57:21.797 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:21.797 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.797 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:57:21.798 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:57:21.798 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:57:21.798 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:57:21.798 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.798 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:57:21.798 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3301", 2024-09-13T11:57:21.798 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:21.798 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:21.798 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.798 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:57:21.798 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790", 2024-09-13T11:57:21.798 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:57:21.798 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:21.799 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:21.799 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:21.799 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790/0", 2024-09-13T11:57:21.799 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-13T11:57:21.799 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:57:21.799 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:57:21.799 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:57:21.799 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:21.799 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:21.799 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:21.799 INFO:teuthology.orchestra.run.smithi100.stdout: "feature_map": { 2024-09-13T11:57:21.799 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": [ 2024-09-13T11:57:21.799 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.800 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:21.800 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:21.800 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:57:21.800 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:21.800 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:21.800 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": [ 2024-09-13T11:57:21.800 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.800 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:21.800 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:21.800 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 3 2024-09-13T11:57:21.800 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:21.800 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:57:21.800 INFO:teuthology.orchestra.run.smithi100.stdout: "client": [ 2024-09-13T11:57:21.801 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:57:21.801 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:57:21.801 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:57:21.801 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 5 2024-09-13T11:57:21.801 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:57:21.801 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:57:21.801 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:57:21.801 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false 2024-09-13T11:57:21.801 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:57:21.802 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.761+0000 7f6989a40640 1 -- 172.21.15.100:0/1178896228 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f697003df50 msgr2=0x7f6970040410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:21.803 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.761+0000 7f6989a40640 1 --2- 172.21.15.100:0/1178896228 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f697003df50 0x7f6970040410 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f696c00c430 tx=0x7f697c005000 comp rx=0 tx=0).stop 2024-09-13T11:57:21.803 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.761+0000 7f6989a40640 10 monclient: shutdown 2024-09-13T11:57:21.803 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.761+0000 7f6989a40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:57:21.803 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.761+0000 7f6989a40640 1 -- 172.21.15.100:0/1178896228 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69841cccb0 msgr2=0x7f69841240a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:57:21.803 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.761+0000 7f6989a40640 1 --2- 172.21.15.100:0/1178896228 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69841cccb0 0x7f69841240a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f697400c730 tx=0x7f697400cbf0 comp rx=0 tx=0).stop 2024-09-13T11:57:21.803 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.761+0000 7f69837fe640 1 -- 172.21.15.100:0/1178896228 reap_dead start 2024-09-13T11:57:21.803 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.761+0000 7f69837fe640 1 -- 172.21.15.100:0/1178896228 reap_dead start 2024-09-13T11:57:21.803 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.761+0000 7f6989a40640 1 -- 172.21.15.100:0/1178896228 shutdown_connections 2024-09-13T11:57:21.803 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.761+0000 7f6989a40640 1 -- 172.21.15.100:0/1178896228 >> 172.21.15.100:0/1178896228 conn(0x7f69840aa580 msgr2=0x7f69840a2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:57:21.803 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.761+0000 7f6989a40640 1 -- 172.21.15.100:0/1178896228 shutdown_connections 2024-09-13T11:57:21.803 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:57:21.761+0000 7f6989a40640 1 -- 172.21.15.100:0/1178896228 wait complete. 2024-09-13T11:57:21.804 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-13T11:57:21.804 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-13T11:57:23.219 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:23.219 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38152]: starting. 2024-09-13T11:57:23.219 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38152]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-38117 2024-09-13T11:57:23.219 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38152]: created object 0... 2024-09-13T11:57:23.219 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38152]: created object 25... 2024-09-13T11:57:23.219 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38152]: created object 49... 2024-09-13T11:57:23.219 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38152]: finishing. 2024-09-13T11:57:23.219 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38152]: shutting down. 2024-09-13T11:57:23.231 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:23.231 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38154]: starting. 2024-09-13T11:57:23.231 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38154]: listing objects. 2024-09-13T11:57:23.231 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38154]: listed object 0... 2024-09-13T11:57:23.231 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38154]: listed object 25... 2024-09-13T11:57:23.231 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38154]: saw 50 objects 2024-09-13T11:57:23.231 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38154]: shutting down. 2024-09-13T11:57:24.037 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:24.037 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38153]: starting. 2024-09-13T11:57:24.037 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38153]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-38117 2024-09-13T11:57:24.037 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38153]: shutting down. 2024-09-13T11:57:24.038 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:24.038 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:24.038 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:57:24.041 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:57:26.118 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38221]: starting. 2024-09-13T11:57:26.118 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38221]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-38220 2024-09-13T11:57:26.118 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38221]: created object 0... 2024-09-13T11:57:26.118 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38221]: created object 25... 2024-09-13T11:57:26.119 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38221]: created object 49... 2024-09-13T11:57:26.119 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38221]: finishing. 2024-09-13T11:57:26.119 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38221]: shutting down. 2024-09-13T11:57:27.044 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38222]: starting. 2024-09-13T11:57:27.045 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38222]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-38220 2024-09-13T11:57:27.045 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38222]: shutting down. 2024-09-13T11:57:29.143 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:29.143 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38255]: starting. 2024-09-13T11:57:29.143 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38255]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-38220 2024-09-13T11:57:29.143 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38255]: created object 0... 2024-09-13T11:57:29.143 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38255]: created object 25... 2024-09-13T11:57:29.143 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38255]: created object 49... 2024-09-13T11:57:29.144 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38255]: finishing. 2024-09-13T11:57:29.144 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38255]: shutting down. 2024-09-13T11:57:29.358 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:29.358 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38257]: starting. 2024-09-13T11:57:29.358 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38257]: listing objects. 2024-09-13T11:57:29.358 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38257]: listed object 0... 2024-09-13T11:57:29.358 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38257]: listed object 25... 2024-09-13T11:57:29.358 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38257]: saw 50 objects 2024-09-13T11:57:29.358 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38257]: shutting down. 2024-09-13T11:57:30.070 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:30.071 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38256]: starting. 2024-09-13T11:57:30.071 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38256]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-38220 2024-09-13T11:57:30.071 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38256]: shutting down. 2024-09-13T11:57:30.072 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:30.072 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:30.072 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:57:30.075 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:57:32.174 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38324]: starting. 2024-09-13T11:57:32.174 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38324]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-38323 2024-09-13T11:57:32.174 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38324]: created object 0... 2024-09-13T11:57:32.174 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38324]: created object 25... 2024-09-13T11:57:32.175 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38324]: created object 49... 2024-09-13T11:57:32.175 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38324]: finishing. 2024-09-13T11:57:32.175 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38324]: shutting down. 2024-09-13T11:57:33.088 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38325]: starting. 2024-09-13T11:57:33.088 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38325]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-38323 2024-09-13T11:57:33.088 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38325]: shutting down. 2024-09-13T11:57:35.124 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:35.125 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38358]: starting. 2024-09-13T11:57:35.125 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38358]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-38323 2024-09-13T11:57:35.125 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38358]: created object 0... 2024-09-13T11:57:35.125 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38358]: created object 25... 2024-09-13T11:57:35.125 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38358]: created object 49... 2024-09-13T11:57:35.125 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38358]: finishing. 2024-09-13T11:57:35.125 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38358]: shutting down. 2024-09-13T11:57:35.137 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:35.137 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38360]: starting. 2024-09-13T11:57:35.137 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38360]: listing objects. 2024-09-13T11:57:35.137 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38360]: listed object 0... 2024-09-13T11:57:35.137 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38360]: listed object 25... 2024-09-13T11:57:35.137 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38360]: saw 50 objects 2024-09-13T11:57:35.137 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38360]: shutting down. 2024-09-13T11:57:36.045 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:36.045 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38359]: starting. 2024-09-13T11:57:36.045 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38359]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-38323 2024-09-13T11:57:36.045 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38359]: shutting down. 2024-09-13T11:57:36.046 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:36.046 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:36.047 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:57:36.049 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:57:38.200 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38427]: starting. 2024-09-13T11:57:38.200 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38427]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-38426 2024-09-13T11:57:38.200 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38427]: created object 0... 2024-09-13T11:57:38.200 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38427]: created object 25... 2024-09-13T11:57:38.201 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38427]: created object 49... 2024-09-13T11:57:38.201 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38427]: finishing. 2024-09-13T11:57:38.201 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38427]: shutting down. 2024-09-13T11:57:39.109 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38428]: starting. 2024-09-13T11:57:39.109 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38428]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-38426 2024-09-13T11:57:39.109 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38428]: shutting down. 2024-09-13T11:57:41.197 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:41.197 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38461]: starting. 2024-09-13T11:57:41.197 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38461]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-38426 2024-09-13T11:57:41.197 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38461]: created object 0... 2024-09-13T11:57:41.197 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38461]: created object 25... 2024-09-13T11:57:41.197 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38461]: created object 49... 2024-09-13T11:57:41.197 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38461]: finishing. 2024-09-13T11:57:41.197 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38461]: shutting down. 2024-09-13T11:57:41.209 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:41.209 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38463]: starting. 2024-09-13T11:57:41.209 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38463]: listing objects. 2024-09-13T11:57:41.209 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38463]: listed object 0... 2024-09-13T11:57:41.209 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38463]: listed object 25... 2024-09-13T11:57:41.209 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38463]: saw 50 objects 2024-09-13T11:57:41.209 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38463]: shutting down. 2024-09-13T11:57:42.138 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:42.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38462]: starting. 2024-09-13T11:57:42.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38462]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-38426 2024-09-13T11:57:42.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38462]: shutting down. 2024-09-13T11:57:42.139 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:42.139 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:42.139 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:57:42.142 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:57:44.127 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38530]: starting. 2024-09-13T11:57:44.127 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38530]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-38529 2024-09-13T11:57:44.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38530]: created object 0... 2024-09-13T11:57:44.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38530]: created object 25... 2024-09-13T11:57:44.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38530]: created object 49... 2024-09-13T11:57:44.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38530]: finishing. 2024-09-13T11:57:44.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38530]: shutting down. 2024-09-13T11:57:45.042 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38531]: starting. 2024-09-13T11:57:45.042 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38531]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-38529 2024-09-13T11:57:45.043 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38531]: shutting down. 2024-09-13T11:57:47.148 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:47.148 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38564]: starting. 2024-09-13T11:57:47.148 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38564]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-38529 2024-09-13T11:57:47.148 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38564]: created object 0... 2024-09-13T11:57:47.148 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38564]: created object 25... 2024-09-13T11:57:47.149 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38564]: created object 49... 2024-09-13T11:57:47.149 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38564]: finishing. 2024-09-13T11:57:47.149 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38564]: shutting down. 2024-09-13T11:57:47.160 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:47.160 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38566]: starting. 2024-09-13T11:57:47.160 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38566]: listing objects. 2024-09-13T11:57:47.160 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38566]: listed object 0... 2024-09-13T11:57:47.160 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38566]: listed object 25... 2024-09-13T11:57:47.160 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38566]: saw 50 objects 2024-09-13T11:57:47.161 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38566]: shutting down. 2024-09-13T11:57:48.065 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:48.065 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38565]: starting. 2024-09-13T11:57:48.065 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38565]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-38529 2024-09-13T11:57:48.065 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38565]: shutting down. 2024-09-13T11:57:48.066 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:48.066 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:48.066 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:57:48.069 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:57:50.169 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38633]: starting. 2024-09-13T11:57:50.169 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38633]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-38632 2024-09-13T11:57:50.169 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38633]: created object 0... 2024-09-13T11:57:50.169 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38633]: created object 25... 2024-09-13T11:57:50.169 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38633]: created object 49... 2024-09-13T11:57:50.169 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38633]: finishing. 2024-09-13T11:57:50.169 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38633]: shutting down. 2024-09-13T11:57:51.082 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38634]: starting. 2024-09-13T11:57:51.082 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38634]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-38632 2024-09-13T11:57:51.082 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38634]: shutting down. 2024-09-13T11:57:53.173 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:53.173 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38667]: starting. 2024-09-13T11:57:53.173 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38667]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-38632 2024-09-13T11:57:53.173 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38667]: created object 0... 2024-09-13T11:57:53.173 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38667]: created object 25... 2024-09-13T11:57:53.173 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38667]: created object 49... 2024-09-13T11:57:53.173 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38667]: finishing. 2024-09-13T11:57:53.173 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38667]: shutting down. 2024-09-13T11:57:53.185 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:53.186 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38669]: starting. 2024-09-13T11:57:53.186 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38669]: listing objects. 2024-09-13T11:57:53.186 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38669]: listed object 0... 2024-09-13T11:57:53.186 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38669]: listed object 25... 2024-09-13T11:57:53.186 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38669]: saw 50 objects 2024-09-13T11:57:53.186 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38669]: shutting down. 2024-09-13T11:57:54.045 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:54.045 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38668]: starting. 2024-09-13T11:57:54.045 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38668]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-38632 2024-09-13T11:57:54.045 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38668]: shutting down. 2024-09-13T11:57:54.046 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:54.046 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:54.046 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:57:54.049 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:57:56.196 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38736]: starting. 2024-09-13T11:57:56.196 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38736]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-38735 2024-09-13T11:57:56.196 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38736]: created object 0... 2024-09-13T11:57:56.197 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38736]: created object 25... 2024-09-13T11:57:56.197 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38736]: created object 49... 2024-09-13T11:57:56.197 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38736]: finishing. 2024-09-13T11:57:56.197 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38736]: shutting down. 2024-09-13T11:57:57.117 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38737]: starting. 2024-09-13T11:57:57.117 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38737]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-38735 2024-09-13T11:57:57.117 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38737]: shutting down. 2024-09-13T11:57:59.209 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:59.210 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38770]: starting. 2024-09-13T11:57:59.210 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38770]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-38735 2024-09-13T11:57:59.210 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38770]: created object 0... 2024-09-13T11:57:59.210 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38770]: created object 25... 2024-09-13T11:57:59.210 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38770]: created object 49... 2024-09-13T11:57:59.210 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38770]: finishing. 2024-09-13T11:57:59.210 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38770]: shutting down. 2024-09-13T11:57:59.222 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:57:59.222 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38772]: starting. 2024-09-13T11:57:59.222 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38772]: listing objects. 2024-09-13T11:57:59.222 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38772]: listed object 0... 2024-09-13T11:57:59.222 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38772]: listed object 25... 2024-09-13T11:57:59.222 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38772]: saw 50 objects 2024-09-13T11:57:59.222 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38772]: shutting down. 2024-09-13T11:58:00.138 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:00.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38771]: starting. 2024-09-13T11:58:00.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38771]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-38735 2024-09-13T11:58:00.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38771]: shutting down. 2024-09-13T11:58:00.139 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:00.139 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:00.139 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:58:00.142 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:58:02.229 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38840]: starting. 2024-09-13T11:58:02.229 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38840]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-38839 2024-09-13T11:58:02.229 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38840]: created object 0... 2024-09-13T11:58:02.229 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38840]: created object 25... 2024-09-13T11:58:02.229 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38840]: created object 49... 2024-09-13T11:58:02.229 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38840]: finishing. 2024-09-13T11:58:02.229 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38840]: shutting down. 2024-09-13T11:58:03.148 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38841]: starting. 2024-09-13T11:58:03.148 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38841]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-38839 2024-09-13T11:58:03.149 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38841]: shutting down. 2024-09-13T11:58:05.193 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:05.194 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38874]: starting. 2024-09-13T11:58:05.194 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38874]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-38839 2024-09-13T11:58:05.194 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38874]: created object 0... 2024-09-13T11:58:05.194 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38874]: created object 25... 2024-09-13T11:58:05.194 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38874]: created object 49... 2024-09-13T11:58:05.194 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38874]: finishing. 2024-09-13T11:58:05.194 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38874]: shutting down. 2024-09-13T11:58:05.206 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:05.206 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38876]: starting. 2024-09-13T11:58:05.206 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38876]: listing objects. 2024-09-13T11:58:05.207 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38876]: listed object 0... 2024-09-13T11:58:05.207 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38876]: listed object 25... 2024-09-13T11:58:05.207 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38876]: saw 50 objects 2024-09-13T11:58:05.207 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38876]: shutting down. 2024-09-13T11:58:06.049 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:06.050 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38875]: starting. 2024-09-13T11:58:06.050 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38875]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-38839 2024-09-13T11:58:06.050 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38875]: shutting down. 2024-09-13T11:58:06.051 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:06.052 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:06.052 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:58:06.054 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:58:08.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38944]: starting. 2024-09-13T11:58:08.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38944]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-38943 2024-09-13T11:58:08.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38944]: created object 0... 2024-09-13T11:58:08.129 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38944]: created object 25... 2024-09-13T11:58:08.129 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38944]: created object 49... 2024-09-13T11:58:08.129 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38944]: finishing. 2024-09-13T11:58:08.129 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[38944]: shutting down. 2024-09-13T11:58:09.054 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38945]: starting. 2024-09-13T11:58:09.054 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38945]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-38943 2024-09-13T11:58:09.055 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[38945]: shutting down. 2024-09-13T11:58:11.137 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:11.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38978]: starting. 2024-09-13T11:58:11.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38978]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-38943 2024-09-13T11:58:11.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38978]: created object 0... 2024-09-13T11:58:11.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38978]: created object 25... 2024-09-13T11:58:11.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38978]: created object 49... 2024-09-13T11:58:11.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38978]: finishing. 2024-09-13T11:58:11.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[38978]: shutting down. 2024-09-13T11:58:11.149 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:11.150 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38980]: starting. 2024-09-13T11:58:11.150 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38980]: listing objects. 2024-09-13T11:58:11.150 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38980]: listed object 0... 2024-09-13T11:58:11.150 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38980]: listed object 25... 2024-09-13T11:58:11.150 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38980]: saw 50 objects 2024-09-13T11:58:11.150 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[38980]: shutting down. 2024-09-13T11:58:12.057 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:12.057 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38979]: starting. 2024-09-13T11:58:12.057 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38979]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-38943 2024-09-13T11:58:12.057 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[38979]: shutting down. 2024-09-13T11:58:12.059 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:12.059 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:12.059 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:58:12.061 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:58:14.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39047]: starting. 2024-09-13T11:58:14.128 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39047]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-39046 2024-09-13T11:58:14.129 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39047]: created object 0... 2024-09-13T11:58:14.129 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39047]: created object 25... 2024-09-13T11:58:14.129 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39047]: created object 49... 2024-09-13T11:58:14.129 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39047]: finishing. 2024-09-13T11:58:14.129 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39047]: shutting down. 2024-09-13T11:58:15.051 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[39048]: starting. 2024-09-13T11:58:15.052 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[39048]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-39046 2024-09-13T11:58:15.052 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[39048]: shutting down. 2024-09-13T11:58:17.139 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:17.139 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39081]: starting. 2024-09-13T11:58:17.139 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39081]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-39046 2024-09-13T11:58:17.139 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39081]: created object 0... 2024-09-13T11:58:17.139 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39081]: created object 25... 2024-09-13T11:58:17.139 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39081]: created object 49... 2024-09-13T11:58:17.139 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39081]: finishing. 2024-09-13T11:58:17.139 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39081]: shutting down. 2024-09-13T11:58:17.151 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:17.151 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39083]: starting. 2024-09-13T11:58:17.151 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39083]: listing objects. 2024-09-13T11:58:17.151 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39083]: listed object 0... 2024-09-13T11:58:17.151 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39083]: listed object 25... 2024-09-13T11:58:17.151 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39083]: saw 50 objects 2024-09-13T11:58:17.152 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39083]: shutting down. 2024-09-13T11:58:18.067 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:18.067 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[39082]: starting. 2024-09-13T11:58:18.067 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[39082]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-39046 2024-09-13T11:58:18.067 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[39082]: shutting down. 2024-09-13T11:58:18.069 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:18.069 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:18.069 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:58:18.072 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:58:20.163 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39150]: starting. 2024-09-13T11:58:20.164 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39150]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-39149 2024-09-13T11:58:20.164 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39150]: created object 0... 2024-09-13T11:58:20.164 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39150]: created object 25... 2024-09-13T11:58:20.164 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39150]: created object 49... 2024-09-13T11:58:20.164 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39150]: finishing. 2024-09-13T11:58:20.164 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39150]: shutting down. 2024-09-13T11:58:21.083 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[39151]: starting. 2024-09-13T11:58:21.083 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[39151]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-39149 2024-09-13T11:58:21.083 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[39151]: shutting down. 2024-09-13T11:58:23.172 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:23.173 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39184]: starting. 2024-09-13T11:58:23.173 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39184]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-39149 2024-09-13T11:58:23.173 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39184]: created object 0... 2024-09-13T11:58:23.173 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39184]: created object 25... 2024-09-13T11:58:23.173 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39184]: created object 49... 2024-09-13T11:58:23.173 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39184]: finishing. 2024-09-13T11:58:23.173 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39184]: shutting down. 2024-09-13T11:58:23.185 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:23.185 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39186]: starting. 2024-09-13T11:58:23.185 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39186]: listing objects. 2024-09-13T11:58:23.185 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39186]: listed object 0... 2024-09-13T11:58:23.185 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39186]: listed object 25... 2024-09-13T11:58:23.185 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39186]: saw 50 objects 2024-09-13T11:58:23.185 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39186]: shutting down. 2024-09-13T11:58:24.050 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:24.050 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[39185]: starting. 2024-09-13T11:58:24.050 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[39185]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-39149 2024-09-13T11:58:24.050 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[39185]: shutting down. 2024-09-13T11:58:24.051 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:24.051 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:24.051 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:58:24.054 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:58:26.204 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39253]: starting. 2024-09-13T11:58:26.204 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39253]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-39252 2024-09-13T11:58:26.204 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39253]: created object 0... 2024-09-13T11:58:26.204 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39253]: created object 25... 2024-09-13T11:58:26.204 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39253]: created object 49... 2024-09-13T11:58:26.204 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39253]: finishing. 2024-09-13T11:58:26.204 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39253]: shutting down. 2024-09-13T11:58:27.121 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[39254]: starting. 2024-09-13T11:58:27.121 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[39254]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-39252 2024-09-13T11:58:27.121 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[39254]: shutting down. 2024-09-13T11:58:29.211 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:29.211 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39287]: starting. 2024-09-13T11:58:29.212 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39287]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-39252 2024-09-13T11:58:29.212 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39287]: created object 0... 2024-09-13T11:58:29.212 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39287]: created object 25... 2024-09-13T11:58:29.212 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39287]: created object 49... 2024-09-13T11:58:29.212 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39287]: finishing. 2024-09-13T11:58:29.212 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39287]: shutting down. 2024-09-13T11:58:29.223 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:29.223 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39289]: starting. 2024-09-13T11:58:29.223 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39289]: listing objects. 2024-09-13T11:58:29.224 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39289]: listed object 0... 2024-09-13T11:58:29.224 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39289]: listed object 25... 2024-09-13T11:58:29.224 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39289]: saw 50 objects 2024-09-13T11:58:29.224 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39289]: shutting down. 2024-09-13T11:58:30.137 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:30.137 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[39288]: starting. 2024-09-13T11:58:30.137 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[39288]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-39252 2024-09-13T11:58:30.137 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[39288]: shutting down. 2024-09-13T11:58:30.138 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:30.138 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:30.138 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:58:30.141 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:58:32.228 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39356]: starting. 2024-09-13T11:58:32.229 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39356]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-39355 2024-09-13T11:58:32.229 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39356]: created object 0... 2024-09-13T11:58:32.229 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39356]: created object 25... 2024-09-13T11:58:32.229 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39356]: created object 49... 2024-09-13T11:58:32.229 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39356]: finishing. 2024-09-13T11:58:32.229 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39356]: shutting down. 2024-09-13T11:58:33.150 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[39357]: starting. 2024-09-13T11:58:33.150 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[39357]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-39355 2024-09-13T11:58:33.150 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[39357]: shutting down. 2024-09-13T11:58:35.138 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:35.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39390]: starting. 2024-09-13T11:58:35.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39390]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-39355 2024-09-13T11:58:35.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39390]: created object 0... 2024-09-13T11:58:35.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39390]: created object 25... 2024-09-13T11:58:35.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39390]: created object 49... 2024-09-13T11:58:35.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39390]: finishing. 2024-09-13T11:58:35.138 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39390]: shutting down. 2024-09-13T11:58:35.150 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:35.150 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39392]: starting. 2024-09-13T11:58:35.150 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39392]: listing objects. 2024-09-13T11:58:35.150 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39392]: listed object 0... 2024-09-13T11:58:35.150 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39392]: listed object 25... 2024-09-13T11:58:35.150 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39392]: saw 50 objects 2024-09-13T11:58:35.150 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39392]: shutting down. 2024-09-13T11:58:36.056 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:36.056 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[39391]: starting. 2024-09-13T11:58:36.056 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[39391]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-39355 2024-09-13T11:58:36.056 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[39391]: shutting down. 2024-09-13T11:58:36.057 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:36.057 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:36.058 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:58:36.060 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:58:38.132 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39459]: starting. 2024-09-13T11:58:38.132 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39459]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-39458 2024-09-13T11:58:38.132 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39459]: created object 0... 2024-09-13T11:58:38.132 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39459]: created object 25... 2024-09-13T11:58:38.133 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39459]: created object 49... 2024-09-13T11:58:38.133 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39459]: finishing. 2024-09-13T11:58:38.133 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39459]: shutting down. 2024-09-13T11:58:39.061 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[39460]: starting. 2024-09-13T11:58:39.061 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[39460]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-39458 2024-09-13T11:58:39.061 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[39460]: shutting down. 2024-09-13T11:58:41.147 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:41.147 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39493]: starting. 2024-09-13T11:58:41.147 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39493]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-39458 2024-09-13T11:58:41.147 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39493]: created object 0... 2024-09-13T11:58:41.147 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39493]: created object 25... 2024-09-13T11:58:41.147 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39493]: created object 49... 2024-09-13T11:58:41.147 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39493]: finishing. 2024-09-13T11:58:41.147 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39493]: shutting down. 2024-09-13T11:58:41.158 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:41.158 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39495]: starting. 2024-09-13T11:58:41.158 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39495]: listing objects. 2024-09-13T11:58:41.158 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39495]: listed object 0... 2024-09-13T11:58:41.159 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39495]: listed object 25... 2024-09-13T11:58:41.159 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39495]: saw 50 objects 2024-09-13T11:58:41.159 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39495]: shutting down. 2024-09-13T11:58:42.071 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:42.072 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[39494]: starting. 2024-09-13T11:58:42.072 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[39494]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-39458 2024-09-13T11:58:42.072 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[39494]: shutting down. 2024-09-13T11:58:42.073 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:42.073 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:42.073 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:58:42.076 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-13T11:58:44.139 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39562]: starting. 2024-09-13T11:58:44.140 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39562]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-39561 2024-09-13T11:58:44.140 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39562]: created object 0... 2024-09-13T11:58:44.140 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39562]: created object 25... 2024-09-13T11:58:44.140 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39562]: created object 49... 2024-09-13T11:58:44.140 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39562]: finishing. 2024-09-13T11:58:44.140 INFO:teuthology.orchestra.run.smithi122.stdout:process_1_[39562]: shutting down. 2024-09-13T11:58:45.059 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[39563]: starting. 2024-09-13T11:58:45.059 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[39563]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-39561 2024-09-13T11:58:45.059 INFO:teuthology.orchestra.run.smithi122.stdout:process_2_[39563]: shutting down. 2024-09-13T11:58:47.150 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:47.151 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39596]: starting. 2024-09-13T11:58:47.151 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39596]: creating pool ceph_test_rados_delete_pools_parallel.smithi122-39561 2024-09-13T11:58:47.151 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39596]: created object 0... 2024-09-13T11:58:47.151 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39596]: created object 25... 2024-09-13T11:58:47.151 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39596]: created object 49... 2024-09-13T11:58:47.151 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39596]: finishing. 2024-09-13T11:58:47.151 INFO:teuthology.orchestra.run.smithi122.stdout:process_3_[39596]: shutting down. 2024-09-13T11:58:47.162 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:47.163 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39598]: starting. 2024-09-13T11:58:47.163 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39598]: listing objects. 2024-09-13T11:58:47.163 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39598]: listed object 0... 2024-09-13T11:58:47.163 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39598]: listed object 25... 2024-09-13T11:58:47.163 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39598]: saw 50 objects 2024-09-13T11:58:47.163 INFO:teuthology.orchestra.run.smithi122.stdout:process_5_[39598]: shutting down. 2024-09-13T11:58:48.073 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:48.074 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[39597]: starting. 2024-09-13T11:58:48.074 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[39597]: deleting pool ceph_test_rados_delete_pools_parallel.smithi122-39561 2024-09-13T11:58:48.074 INFO:teuthology.orchestra.run.smithi122.stdout:process_4_[39597]: shutting down. 2024-09-13T11:58:48.075 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:48.075 INFO:teuthology.orchestra.run.smithi122.stdout:******************************* 2024-09-13T11:58:48.075 INFO:teuthology.orchestra.run.smithi122.stdout:******* SUCCESS ********** 2024-09-13T11:58:48.077 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-13T11:58:48.086 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-13T11:58:51.805 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-13T11:58:51.805 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-13T11:58:51.805 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-13T11:58:51.808 INFO:tasks.ceph.mon.b:Started 2024-09-13T11:58:51.808 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-13T11:58:51.808 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:58:51.905 INFO:tasks.ceph.mon.b.smithi122.stderr:2024-09-13T11:58:51.903+0000 7f001732ec80 -1 mon.b@-1(???).log v0 log_external_backlog rewinding external_log_to from 217 -> 0 (sync_force? mon rebuild?) 2024-09-13T11:58:51.915 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.914+0000 7f55b16be640 10 monclient: get_monmap_and_config 2024-09-13T11:58:51.915 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.914+0000 7f55b16be640 10 monclient: build_initial_monmap 2024-09-13T11:58:51.915 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.914+0000 7f55b16be640 10 monclient: monmap: 2024-09-13T11:58:51.915 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:58:51.915 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:58:51.916 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:58:51.915406+0000 2024-09-13T11:58:51.916 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:58:51.915406+0000 2024-09-13T11:58:51.916 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:58:51.916 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:58:51.916 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:58:51.916 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:58:51.916 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:58:51.916 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:58:51.916 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:58:51.916 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:58:51.924 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55b16be640 1 Processor -- start 2024-09-13T11:58:51.925 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55b16be640 1 -- start start 2024-09-13T11:58:51.925 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55b16be640 10 monclient: init 2024-09-13T11:58:51.925 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55b16be640 10 monclient: _reopen_session rank -1 2024-09-13T11:58:51.925 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55b16be640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-13T11:58:51.925 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55b16be640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f55ac0a3440 0x7f55ac0a3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:58:51.925 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55b16be640 10 monclient(hunting): picked mon.noname-b con 0x7f55ac0a3440 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:58:51.925 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55b16be640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f55ac0a28a0 addr v1:172.21.15.100:6789/0 2024-09-13T11:58:51.925 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55b16be640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f55ac0a4ff0 addr v1:172.21.15.122:6789/0 2024-09-13T11:58:51.925 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55b16be640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55ac0a3d60 con 0x7f55ac0a28a0 2024-09-13T11:58:51.925 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55b16be640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55ac0a67b0 con 0x7f55ac0a4ff0 2024-09-13T11:58:51.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55b16be640 10 monclient(hunting): start opening mon connection 2024-09-13T11:58:51.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55b16be640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f55ac0a9d20 con 0x7f55ac0a3440 2024-09-13T11:58:51.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55b16be640 10 monclient(hunting): _renew_subs 2024-09-13T11:58:51.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55b16be640 10 monclient(hunting): authenticate will time out at 1112.508732s 2024-09-13T11:58:51.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55aa7fc640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f55ac0a28a0 0x7f55ac0a48e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:51762/0 (socket says 172.21.15.100:51762) 2024-09-13T11:58:51.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55aaffd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f55ac0a3440 0x7f55ac0a3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:58:51.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55aa7fc640 1 -- 172.21.15.100:0/1576928737 learned_addr learned my addr 172.21.15.100:0/1576928737 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:58:51.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55aaffd640 10 monclient(hunting): get_auth_request con 0x7f55ac0a3440 auth_method 0 2024-09-13T11:58:51.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:58:51.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:58:51.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:58:51.926 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55aaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:58:51.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55aaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:58:51.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.915+0000 7f55aaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:58:51.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.916+0000 7f55a9ffb640 1 -- 172.21.15.100:0/1576928737 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4235137671 0 0) 0x7f55a0001110 con 0x7f55ac0a28a0 2024-09-13T11:58:51.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.916+0000 7f55a9ffb640 10 monclient(hunting): _negotiate 2024-09-13T11:58:51.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.916+0000 7f55a9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:58:51.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.916+0000 7f55a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:58:51.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.916+0000 7f55a9ffb640 10 monclient(hunting): my global_id is 15441 2024-09-13T11:58:51.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.916+0000 7f55a9ffb640 1 -- 172.21.15.100:0/1576928737 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5594003520 con 0x7f55ac0a28a0 2024-09-13T11:58:51.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.916+0000 7f55aaffd640 10 monclient(hunting): handle_auth_done global_id 24548 payload 274 2024-09-13T11:58:51.927 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.916+0000 7f55aaffd640 1 -- 172.21.15.100:0/1576928737 >> v1:172.21.15.122:6789/0 conn(0x7f55ac0a4ff0 legacy=0x7f55ac13f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:58:51.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.916+0000 7f55aaffd640 1 -- 172.21.15.100:0/1576928737 >> v1:172.21.15.100:6789/0 conn(0x7f55ac0a28a0 legacy=0x7f55ac0a48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:58:51.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.916+0000 7f55aaffd640 10 monclient: _finish_hunting 0 2024-09-13T11:58:51.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.916+0000 7f55aaffd640 1 monclient: found mon.noname-b 2024-09-13T11:58:51.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.916+0000 7f55aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:58:51.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.916+0000 7f55aaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:58:51.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.916+0000 7f55aaffd640 1 -- 172.21.15.100:0/1576928737 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55ac0a1620 con 0x7f55ac0a3440 2024-09-13T11:58:51.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.916+0000 7f55aaffd640 1 --2- 172.21.15.100:0/1576928737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f55ac0a3440 0x7f55ac0a3820 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f559c0097d0 tx=0x7f559c02a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d2cf0956e1a6b4b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:58:51.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.916+0000 7f55a9ffb640 1 -- 172.21.15.100:0/1576928737 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c037070 con 0x7f55ac0a3440 2024-09-13T11:58:51.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.916+0000 7f55a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:58:51.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.916+0000 7f55a9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:58:51.928 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.916+0000 7f55a9ffb640 10 monclient: dump: 2024-09-13T11:58:51.928 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:58:51.929 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:58:51.929 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:58:51.929 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:58:51.929 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:58:51.929 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:58:51.929 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:58:51.929 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:58:51.929 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:58:51.929 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:58:51.929 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.917+0000 7f55a9ffb640 10 monclient: _finish_auth 0 2024-09-13T11:58:51.929 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.917+0000 7f55a9ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:58:51.929 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.917+0000 7f55a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:58:51.929 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.917+0000 7f55a9ffb640 1 -- 172.21.15.100:0/1576928737 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f559c02ae50 con 0x7f55ac0a3440 2024-09-13T11:58:51.930 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.917+0000 7f55a9ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:58:51.930 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.917+0000 7f55a9ffb640 1 -- 172.21.15.100:0/1576928737 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c032660 con 0x7f55ac0a3440 2024-09-13T11:58:51.930 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.917+0000 7f55b16be640 5 monclient: authenticate success, global_id 24548 2024-09-13T11:58:51.930 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.917+0000 7f55b16be640 10 monclient: get_monmap_and_config success 2024-09-13T11:58:51.930 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.917+0000 7f55b16be640 4 set_mon_vals no callback set 2024-09-13T11:58:51.930 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.917+0000 7f55a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:58:51.930 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.917+0000 7f55a9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:58:51.930 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.917+0000 7f55a9ffb640 10 monclient: dump: 2024-09-13T11:58:51.930 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:58:51.930 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:58:51.930 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:58:51.930 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:58:51.930 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:58:51.931 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:58:51.931 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:58:51.931 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:58:51.931 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:58:51.931 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:58:51.931 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.917+0000 7f55b16be640 10 monclient: shutdown 2024-09-13T11:58:51.931 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.917+0000 7f55b16be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:58:51.931 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.917+0000 7f55b16be640 1 -- 172.21.15.100:0/1576928737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f55ac0a3440 msgr2=0x7f55ac0a3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:58:51.931 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.917+0000 7f55b16be640 1 --2- 172.21.15.100:0/1576928737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f55ac0a3440 0x7f55ac0a3820 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f559c0097d0 tx=0x7f559c02a100 comp rx=0 tx=0).stop 2024-09-13T11:58:51.931 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.917+0000 7f55b16be640 1 -- 172.21.15.100:0/1576928737 shutdown_connections 2024-09-13T11:58:51.931 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.917+0000 7f55b16be640 1 --2- 172.21.15.100:0/1576928737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f55ac0a3440 0x7f55ac0a3820 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:58:51.931 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.917+0000 7f55b16be640 1 -- 172.21.15.100:0/1576928737 >> 172.21.15.100:0/1576928737 conn(0x7f55ac0a6e00 msgr2=0x7f55ac0a7210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:58:51.932 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.917+0000 7f55b16be640 1 -- 172.21.15.100:0/1576928737 shutdown_connections 2024-09-13T11:58:51.932 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.917+0000 7f55b16be640 1 -- 172.21.15.100:0/1576928737 wait complete. 2024-09-13T11:58:51.932 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.918+0000 7f55b16be640 10 monclient: build_initial_monmap 2024-09-13T11:58:51.932 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.918+0000 7f55b16be640 10 monclient: monmap: 2024-09-13T11:58:51.932 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:58:51.932 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:58:51.932 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:58:51.932 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:58:51.932 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:58:51.932 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:58:51.932 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:58:51.932 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:58:51.932 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:58:51.933 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:58:51.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.918+0000 7f55b16be640 1 Processor -- start 2024-09-13T11:58:51.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.918+0000 7f55b16be640 1 -- start start 2024-09-13T11:58:51.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.918+0000 7f55b16be640 10 monclient: init 2024-09-13T11:58:51.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55b16be640 10 monclient: _reopen_session rank -1 2024-09-13T11:58:51.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55b16be640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:58:51.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55b16be640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f55ac0a28a0 0x7f55ac1c07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:58:51.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55b16be640 10 monclient(hunting): picked mon.noname-a con 0x7f55ac0a28a0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:58:51.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55b16be640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55ac0a3440 0x7f55ac1aa0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:58:51.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55b16be640 10 monclient(hunting): picked mon.noname-b con 0x7f55ac0a3440 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:58:51.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55b16be640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f55ac0a4ff0 0x7f55ac1aa630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:58:51.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55b16be640 10 monclient(hunting): picked mon.noname-c con 0x7f55ac0a4ff0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:58:51.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55b16be640 10 monclient(hunting): start opening mon connection 2024-09-13T11:58:51.934 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55b16be640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f55ac0a98e0 con 0x7f55ac0a28a0 2024-09-13T11:58:51.934 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55b16be640 10 monclient(hunting): start opening mon connection 2024-09-13T11:58:51.934 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55b16be640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f55ac0a9760 con 0x7f55ac0a3440 2024-09-13T11:58:51.934 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55b16be640 10 monclient(hunting): start opening mon connection 2024-09-13T11:58:51.934 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55b16be640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f55ac0a2ca0 con 0x7f55ac0a4ff0 2024-09-13T11:58:51.934 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55b16be640 10 monclient(hunting): _renew_subs 2024-09-13T11:58:51.934 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55b16be640 10 monclient(hunting): authenticate will time out at 1112.512289s 2024-09-13T11:58:51.934 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aaffd640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f55ac0a28a0 0x7f55ac1c07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:58:51.934 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aaffd640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f55ac0a28a0 0x7f55ac1c07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:51454/0 (socket says 172.21.15.100:51454) 2024-09-13T11:58:51.934 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aaffd640 1 -- 172.21.15.100:0/4180960718 learned_addr learned my addr 172.21.15.100:0/4180960718 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:58:51.934 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aaffd640 10 monclient(hunting): get_auth_request con 0x7f55ac0a28a0 auth_method 0 2024-09-13T11:58:51.935 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:58:51.935 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:58:51.935 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:58:51.935 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55ab7fe640 1 --2- 172.21.15.100:0/4180960718 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f55ac0a4ff0 0x7f55ac1aa630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:58:51.935 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55ab7fe640 10 monclient(hunting): get_auth_request con 0x7f55ac0a4ff0 auth_method 0 2024-09-13T11:58:51.935 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:58:51.935 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:58:51.935 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:58:51.935 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:58:51.935 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:58:51.935 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:58:51.935 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:58:51.935 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:58:51.936 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aa7fc640 1 --2- 172.21.15.100:0/4180960718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55ac0a3440 0x7f55ac1aa0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:58:51.936 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:58:51.936 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aa7fc640 10 monclient(hunting): get_auth_request con 0x7f55ac0a3440 auth_method 0 2024-09-13T11:58:51.936 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:58:51.936 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:58:51.936 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:58:51.936 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aa7fc640 1 -- 172.21.15.100:0/4180960718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55ac0a3440 msgr2=0x7f55ac1aa0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-13T11:58:51.936 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aa7fc640 1 -- 172.21.15.100:0/4180960718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55ac0a3440 msgr2=0x7f55ac1aa0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T11:58:51.936 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aa7fc640 1 --2- 172.21.15.100:0/4180960718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55ac0a3440 0x7f55ac1aa0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:58:51.936 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aa7fc640 1 --2- 172.21.15.100:0/4180960718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55ac0a3440 0x7f55ac1aa0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:58:51.936 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aaffd640 10 monclient(hunting): handle_auth_done global_id 15444 payload 824 2024-09-13T11:58:51.936 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aaffd640 1 -- 172.21.15.100:0/4180960718 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f55ac0a4ff0 msgr2=0x7f55ac1aa630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:58:51.936 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aaffd640 1 --2- 172.21.15.100:0/4180960718 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f55ac0a4ff0 0x7f55ac1aa630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:58:51.937 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aaffd640 1 -- 172.21.15.100:0/4180960718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55ac0a3440 msgr2=0x7f55ac1aa0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:58:51.937 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aaffd640 1 --2- 172.21.15.100:0/4180960718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55ac0a3440 0x7f55ac1aa0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:58:51.937 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aaffd640 10 monclient: _finish_hunting 0 2024-09-13T11:58:51.937 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aaffd640 1 monclient: found mon.noname-a 2024-09-13T11:58:51.937 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:58:51.937 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:58:51.937 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aaffd640 1 -- 172.21.15.100:0/4180960718 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55ac0706f0 con 0x7f55ac0a28a0 2024-09-13T11:58:51.937 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55ab7fe640 1 --2- 172.21.15.100:0/4180960718 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f55ac0a4ff0 0x7f55ac1aa630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:58:51.937 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.919+0000 7f55aaffd640 1 --2- 172.21.15.100:0/4180960718 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f55ac0a28a0 0x7f55ac1c07c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f559c0098f0 tx=0x7f559c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:58:51.937 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.920+0000 7f55937fe640 1 -- 172.21.15.100:0/4180960718 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c041020 con 0x7f55ac0a28a0 2024-09-13T11:58:51.937 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.920+0000 7f55937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:58:51.937 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.920+0000 7f55937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:58:51.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.920+0000 7f55937fe640 10 monclient: dump: 2024-09-13T11:58:51.938 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:58:51.938 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:58:51.938 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:58:51.938 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:58:51.938 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:58:51.938 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:58:51.938 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:58:51.938 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:58:51.938 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:58:51.938 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:58:51.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.920+0000 7f55937fe640 10 monclient: _finish_auth 0 2024-09-13T11:58:51.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.920+0000 7f55937fe640 10 monclient: _check_auth_tickets 2024-09-13T11:58:51.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.920+0000 7f55937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:58:51.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.920+0000 7f55937fe640 1 -- 172.21.15.100:0/4180960718 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f559c03c670 con 0x7f55ac0a28a0 2024-09-13T11:58:51.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.920+0000 7f55b16be640 5 monclient: authenticate success, global_id 15444 2024-09-13T11:58:51.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.920+0000 7f55b16be640 10 monclient: _renew_subs 2024-09-13T11:58:51.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.920+0000 7f55b16be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:58:51.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.920+0000 7f55b16be640 1 -- 172.21.15.100:0/4180960718 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55ac1ad950 con 0x7f55ac0a28a0 2024-09-13T11:58:51.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.920+0000 7f55937fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:58:51.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.920+0000 7f55937fe640 1 -- 172.21.15.100:0/4180960718 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c037040 con 0x7f55ac0a28a0 2024-09-13T11:58:51.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.920+0000 7f55937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:58:51.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.920+0000 7f55937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:58:51.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.920+0000 7f55937fe640 10 monclient: dump: 2024-09-13T11:58:51.939 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:58:51.940 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:58:51.940 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:58:51.940 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:58:51.940 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:58:51.940 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:58:51.940 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:58:51.940 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:58:51.940 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:58:51.940 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:58:51.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.920+0000 7f55b16be640 10 monclient: _renew_subs 2024-09-13T11:58:51.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.920+0000 7f55b16be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:58:51.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.920+0000 7f55a97fa640 4 set_mon_vals no callback set 2024-09-13T11:58:51.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.920+0000 7f55b16be640 1 -- 172.21.15.100:0/4180960718 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55ac0a0590 con 0x7f55ac0a28a0 2024-09-13T11:58:51.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.921+0000 7f55937fe640 1 -- 172.21.15.100:0/4180960718 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f559c044020 con 0x7f55ac0a28a0 2024-09-13T11:58:51.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.921+0000 7f55b16be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:58:51.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.921+0000 7f55b16be640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:58:51.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.921+0000 7f55b16be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:58:51.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.921+0000 7f55b16be640 1 -- 172.21.15.100:0/4180960718 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55ac0a5530 con 0x7f55ac0a28a0 2024-09-13T11:58:51.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.921+0000 7f55937fe640 1 --2- 172.21.15.100:0/4180960718 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f559803dfa0 0x7f5598040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:58:51.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.921+0000 7f55937fe640 1 -- 172.21.15.100:0/4180960718 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7f559c072de0 con 0x7f55ac0a28a0 2024-09-13T11:58:51.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.921+0000 7f55aa7fc640 1 --2- 172.21.15.100:0/4180960718 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f559803dfa0 0x7f5598040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:58:51.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.921+0000 7f55aa7fc640 10 monclient: get_auth_request con 0x7f559803dfa0 auth_method 0 2024-09-13T11:58:51.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.922+0000 7f55aa7fc640 1 --2- 172.21.15.100:0/4180960718 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f559803dfa0 0x7f5598040460 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f55a0001410 tx=0x7f55a0009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:58:51.942 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.926+0000 7f55937fe640 1 -- 172.21.15.100:0/4180960718 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f559c04b030 con 0x7f55ac0a28a0 2024-09-13T11:58:51.942 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.926+0000 7f55937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:58:51.942 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:51.926+0000 7f55937fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:58:52.170 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:52.168+0000 7f55b16be640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:58:52.170 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:52.168+0000 7f55b16be640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:58:52.170 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:52.168+0000 7f55b16be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:58:52.170 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:52.168+0000 7f55b16be640 1 -- 172.21.15.100:0/4180960718 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f55ac0a5b30 con 0x7f55ac0a28a0 2024-09-13T11:58:52.170 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:52.168+0000 7f55937fe640 1 -- 172.21.15.100:0/4180960718 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f55ac0706f0 con 0x7f55ac0a28a0 2024-09-13T11:58:52.170 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:52.168+0000 7f55937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:58:52.170 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:52.168+0000 7f55937fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:58:52.170 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":26,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:58:52.172 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:52.170+0000 7f55b16be640 1 -- 172.21.15.100:0/4180960718 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f559803dfa0 msgr2=0x7f5598040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:58:52.172 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:52.170+0000 7f55b16be640 1 --2- 172.21.15.100:0/4180960718 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f559803dfa0 0x7f5598040460 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f55a0001410 tx=0x7f55a0009040 comp rx=0 tx=0).stop 2024-09-13T11:58:52.172 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:52.170+0000 7f55b16be640 10 monclient: shutdown 2024-09-13T11:58:52.172 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:52.170+0000 7f55b16be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:58:52.172 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:52.170+0000 7f55b16be640 1 -- 172.21.15.100:0/4180960718 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f55ac0a28a0 msgr2=0x7f55ac1c07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:58:52.173 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:52.170+0000 7f55b16be640 1 --2- 172.21.15.100:0/4180960718 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f55ac0a28a0 0x7f55ac1c07c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f559c0098f0 tx=0x7f559c0046c0 comp rx=0 tx=0).stop 2024-09-13T11:58:52.173 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:52.171+0000 7f55b16be640 1 -- 172.21.15.100:0/4180960718 shutdown_connections 2024-09-13T11:58:52.173 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:52.171+0000 7f55b16be640 1 --2- 172.21.15.100:0/4180960718 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f559803dfa0 0x7f5598040460 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:58:52.173 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:52.171+0000 7f55b16be640 1 --2- 172.21.15.100:0/4180960718 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f55ac0a4ff0 0x7f55ac1aa630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:58:52.173 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:52.171+0000 7f55b16be640 1 --2- 172.21.15.100:0/4180960718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55ac0a3440 0x7f55ac1aa0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:58:52.173 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:52.171+0000 7f55b16be640 1 --2- 172.21.15.100:0/4180960718 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f55ac0a28a0 0x7f55ac1c07c0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:58:52.173 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:52.171+0000 7f55b16be640 1 -- 172.21.15.100:0/4180960718 >> 172.21.15.100:0/4180960718 conn(0x7f55ac0a6e00 msgr2=0x7f55ac06f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:58:52.173 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:52.171+0000 7f55b16be640 1 -- 172.21.15.100:0/4180960718 shutdown_connections 2024-09-13T11:58:52.173 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:52.171+0000 7f55b16be640 1 -- 172.21.15.100:0/4180960718 wait complete. 2024-09-13T11:58:55.183 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:58:55.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.289+0000 7f35aaa31640 10 monclient: get_monmap_and_config 2024-09-13T11:58:55.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.289+0000 7f35aaa31640 10 monclient: build_initial_monmap 2024-09-13T11:58:55.290 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.289+0000 7f35aaa31640 10 monclient: monmap: 2024-09-13T11:58:55.291 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:58:55.291 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:58:55.291 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:58:55.290213+0000 2024-09-13T11:58:55.291 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:58:55.290213+0000 2024-09-13T11:58:55.291 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:58:55.291 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:58:55.291 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:58:55.291 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:58:55.291 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:58:55.291 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:58:55.291 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:58:55.291 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:58:55.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35aaa31640 1 Processor -- start 2024-09-13T11:58:55.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35aaa31640 1 -- start start 2024-09-13T11:58:55.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35aaa31640 10 monclient: init 2024-09-13T11:58:55.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35aaa31640 10 monclient: _reopen_session rank -1 2024-09-13T11:58:55.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35aaa31640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-13T11:58:55.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35aaa31640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f35a40a15a0 addr v1:172.21.15.122:6789/0 2024-09-13T11:58:55.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35aaa31640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f35a409f6d0 0x7f35a4139680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:58:55.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35aaa31640 10 monclient(hunting): picked mon.noname-b con 0x7f35a409f6d0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:58:55.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35aaa31640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f35a40a2090 0x7f35a413bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:58:55.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35aaa31640 10 monclient(hunting): picked mon.noname-a con 0x7f35a40a2090 addr v2:172.21.15.100:3300/0 2024-09-13T11:58:55.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35aaa31640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35a409ecf0 con 0x7f35a40a15a0 2024-09-13T11:58:55.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35aaa31640 10 monclient(hunting): start opening mon connection 2024-09-13T11:58:55.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35aaa31640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f35a409d3b0 con 0x7f35a40a2090 2024-09-13T11:58:55.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35aaa31640 10 monclient(hunting): start opening mon connection 2024-09-13T11:58:55.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35aaa31640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f35a409ea10 con 0x7f35a409f6d0 2024-09-13T11:58:55.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35aaa31640 10 monclient(hunting): _renew_subs 2024-09-13T11:58:55.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35aaa31640 10 monclient(hunting): authenticate will time out at 1115.883691s 2024-09-13T11:58:55.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35a8fce640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f35a40a2090 0x7f35a413bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:58:55.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35a37fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f35a409f6d0 0x7f35a4139680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:58:55.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35a8fce640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f35a40a2090 0x7f35a413bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:59996/0 (socket says 172.21.15.100:59996) 2024-09-13T11:58:55.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35a37fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f35a409f6d0 0x7f35a4139680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:51948/0 (socket says 172.21.15.100:51948) 2024-09-13T11:58:55.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35a8fce640 1 -- 172.21.15.100:0/3685778726 learned_addr learned my addr 172.21.15.100:0/3685778726 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:58:55.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35a8fce640 10 monclient(hunting): get_auth_request con 0x7f35a40a2090 auth_method 0 2024-09-13T11:58:55.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35a8fce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:58:55.300 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35a8fce640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:58:55.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35a8fce640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:58:55.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35a37fe640 10 monclient(hunting): get_auth_request con 0x7f35a409f6d0 auth_method 0 2024-09-13T11:58:55.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:58:55.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:58:55.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:58:55.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35a8fce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:58:55.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35a8fce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:58:55.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35a8fce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:58:55.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:58:55.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:58:55.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.290+0000 7f35a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:58:55.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.291+0000 7f35a37fe640 10 monclient(hunting): handle_auth_done global_id 24554 payload 274 2024-09-13T11:58:55.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.291+0000 7f35a37fe640 1 -- 172.21.15.100:0/3685778726 >> v2:172.21.15.100:3300/0 conn(0x7f35a40a2090 msgr2=0x7f35a413bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:58:55.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.291+0000 7f35a37fe640 1 --2- 172.21.15.100:0/3685778726 >> v2:172.21.15.100:3300/0 conn(0x7f35a40a2090 0x7f35a413bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:58:55.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.291+0000 7f35a37fe640 1 -- 172.21.15.100:0/3685778726 >> v1:172.21.15.122:6789/0 conn(0x7f35a40a15a0 legacy=0x7f35a40a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:58:55.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.291+0000 7f35a37fe640 10 monclient: _finish_hunting 0 2024-09-13T11:58:55.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.291+0000 7f35a37fe640 1 monclient: found mon.noname-b 2024-09-13T11:58:55.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.291+0000 7f35a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:58:55.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.291+0000 7f35a37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:58:55.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.291+0000 7f35a37fe640 1 -- 172.21.15.100:0/3685778726 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35a40a3250 con 0x7f35a409f6d0 2024-09-13T11:58:55.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.291+0000 7f35a8fce640 1 --2- 172.21.15.100:0/3685778726 >> v2:172.21.15.100:3300/0 conn(0x7f35a40a2090 0x7f35a413bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:58:55.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.291+0000 7f35a37fe640 1 --2- 172.21.15.100:0/3685778726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f35a409f6d0 0x7f35a4139680 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f35980097a0 tx=0x7f3598029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a00aaf0fbac71a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:58:55.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35a2ffd640 1 -- 172.21.15.100:0/3685778726 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3598037070 con 0x7f35a409f6d0 2024-09-13T11:58:55.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:58:55.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35a2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:58:55.302 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35a2ffd640 10 monclient: dump: 2024-09-13T11:58:55.303 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:58:55.303 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:58:55.303 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:58:55.303 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:58:55.303 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:58:55.303 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:58:55.303 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:58:55.303 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:58:55.303 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:58:55.303 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:58:55.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35a2ffd640 10 monclient: _finish_auth 0 2024-09-13T11:58:55.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35a2ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:58:55.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:58:55.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35a2ffd640 1 -- 172.21.15.100:0/3685778726 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f359802ad20 con 0x7f35a409f6d0 2024-09-13T11:58:55.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35aaa31640 5 monclient: authenticate success, global_id 24554 2024-09-13T11:58:55.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35aaa31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:58:55.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35a2ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:58:55.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35aaa31640 10 monclient: get_monmap_and_config success 2024-09-13T11:58:55.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35a2ffd640 1 -- 172.21.15.100:0/3685778726 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3598032660 con 0x7f35a409f6d0 2024-09-13T11:58:55.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:58:55.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35aaa31640 4 set_mon_vals no callback set 2024-09-13T11:58:55.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35a2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:58:55.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35a2ffd640 10 monclient: dump: 2024-09-13T11:58:55.305 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:58:55.305 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:58:55.305 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:58:55.305 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:58:55.305 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:58:55.305 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:58:55.305 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:58:55.305 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:58:55.305 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:58:55.305 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:58:55.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35aaa31640 10 monclient: shutdown 2024-09-13T11:58:55.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35aaa31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:58:55.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35aaa31640 1 -- 172.21.15.100:0/3685778726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f35a409f6d0 msgr2=0x7f35a4139680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:58:55.306 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35aaa31640 1 --2- 172.21.15.100:0/3685778726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f35a409f6d0 0x7f35a4139680 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f35980097a0 tx=0x7f3598029fd0 comp rx=0 tx=0).stop 2024-09-13T11:58:55.306 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35aaa31640 1 -- 172.21.15.100:0/3685778726 shutdown_connections 2024-09-13T11:58:55.306 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35aaa31640 1 --2- 172.21.15.100:0/3685778726 >> v2:172.21.15.100:3300/0 conn(0x7f35a40a2090 0x7f35a413bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:58:55.306 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35aaa31640 1 --2- 172.21.15.100:0/3685778726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f35a409f6d0 0x7f35a4139680 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:58:55.306 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35aaa31640 1 -- 172.21.15.100:0/3685778726 >> 172.21.15.100:0/3685778726 conn(0x7f35a40a6470 msgr2=0x7f35a40a2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:58:55.306 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35aaa31640 1 -- 172.21.15.100:0/3685778726 shutdown_connections 2024-09-13T11:58:55.306 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.292+0000 7f35aaa31640 1 -- 172.21.15.100:0/3685778726 wait complete. 2024-09-13T11:58:55.306 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.293+0000 7f35aaa31640 10 monclient: build_initial_monmap 2024-09-13T11:58:55.306 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.293+0000 7f35aaa31640 10 monclient: monmap: 2024-09-13T11:58:55.306 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:58:55.306 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:58:55.306 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:58:55.306 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:58:55.307 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:58:55.307 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:58:55.307 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:58:55.307 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:58:55.307 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:58:55.307 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:58:55.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.293+0000 7f35aaa31640 1 Processor -- start 2024-09-13T11:58:55.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.293+0000 7f35aaa31640 1 -- start start 2024-09-13T11:58:55.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.293+0000 7f35aaa31640 10 monclient: init 2024-09-13T11:58:55.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35aaa31640 10 monclient: _reopen_session rank -1 2024-09-13T11:58:55.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35aaa31640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:58:55.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35aaa31640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f35a409f6d0 0x7f35a41cc700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:58:55.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35aaa31640 10 monclient(hunting): picked mon.noname-c con 0x7f35a409f6d0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:58:55.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35aaa31640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f35a40a15a0 0x7f35a41ccc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:58:55.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35aaa31640 10 monclient(hunting): picked mon.noname-a con 0x7f35a40a15a0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:58:55.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35aaa31640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a40a2090 0x7f35a4124f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:58:55.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35aaa31640 10 monclient(hunting): picked mon.noname-b con 0x7f35a40a2090 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:58:55.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35aaa31640 10 monclient(hunting): start opening mon connection 2024-09-13T11:58:55.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35aaa31640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f35a409ba00 con 0x7f35a40a15a0 2024-09-13T11:58:55.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35aaa31640 10 monclient(hunting): start opening mon connection 2024-09-13T11:58:55.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35aaa31640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f35a409d1b0 con 0x7f35a40a2090 2024-09-13T11:58:55.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35aaa31640 10 monclient(hunting): start opening mon connection 2024-09-13T11:58:55.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35aaa31640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f35a409bb80 con 0x7f35a409f6d0 2024-09-13T11:58:55.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35aaa31640 10 monclient(hunting): _renew_subs 2024-09-13T11:58:55.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35aaa31640 10 monclient(hunting): authenticate will time out at 1115.887268s 2024-09-13T11:58:55.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a3fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f35a409f6d0 0x7f35a41cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:58:55.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a37fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f35a40a15a0 0x7f35a41ccc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:58:55.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a37fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f35a40a15a0 0x7f35a41ccc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:60002/0 (socket says 172.21.15.100:60002) 2024-09-13T11:58:55.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a3fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f35a409f6d0 0x7f35a41cc700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:51958/0 (socket says 172.21.15.100:51958) 2024-09-13T11:58:55.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a37fe640 1 -- 172.21.15.100:0/1142489496 learned_addr learned my addr 172.21.15.100:0/1142489496 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:58:55.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a37fe640 10 monclient(hunting): get_auth_request con 0x7f35a40a15a0 auth_method 0 2024-09-13T11:58:55.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:58:55.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:58:55.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:58:55.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a3fff640 10 monclient(hunting): get_auth_request con 0x7f35a409f6d0 auth_method 0 2024-09-13T11:58:55.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:58:55.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:58:55.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:58:55.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:58:55.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:58:55.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:58:55.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:58:55.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:58:55.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:58:55.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a37fe640 10 monclient(hunting): handle_auth_done global_id 15450 payload 824 2024-09-13T11:58:55.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a37fe640 1 -- 172.21.15.100:0/1142489496 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f35a409f6d0 msgr2=0x7f35a41cc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:58:55.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a37fe640 1 --2- 172.21.15.100:0/1142489496 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f35a409f6d0 0x7f35a41cc700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:58:55.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a37fe640 1 -- 172.21.15.100:0/1142489496 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a40a2090 msgr2=0x7f35a4124f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:58:55.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a37fe640 1 --2- 172.21.15.100:0/1142489496 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a40a2090 0x7f35a4124f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:58:55.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a37fe640 10 monclient: _finish_hunting 0 2024-09-13T11:58:55.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a37fe640 1 monclient: found mon.noname-a 2024-09-13T11:58:55.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:58:55.311 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:58:55.311 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a37fe640 1 -- 172.21.15.100:0/1142489496 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35a4128a70 con 0x7f35a40a15a0 2024-09-13T11:58:55.311 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a37fe640 1 --2- 172.21.15.100:0/1142489496 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f35a40a15a0 0x7f35a41ccc40 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3598000c00 tx=0x7f35980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:58:55.311 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a0ff9640 1 -- 172.21.15.100:0/1142489496 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3598037040 con 0x7f35a40a15a0 2024-09-13T11:58:55.311 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:58:55.311 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:58:55.311 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a0ff9640 10 monclient: dump: 2024-09-13T11:58:55.311 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:58:55.311 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:58:55.311 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:58:55.311 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:58:55.311 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:58:55.312 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:58:55.312 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:58:55.312 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:58:55.312 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:58:55.312 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:58:55.312 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a0ff9640 10 monclient: _finish_auth 0 2024-09-13T11:58:55.312 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a0ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:58:55.312 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:58:55.312 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35a0ff9640 1 -- 172.21.15.100:0/1142489496 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3598042070 con 0x7f35a40a15a0 2024-09-13T11:58:55.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35aaa31640 5 monclient: authenticate success, global_id 15450 2024-09-13T11:58:55.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.294+0000 7f35aaa31640 10 monclient: _renew_subs 2024-09-13T11:58:55.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.295+0000 7f35aaa31640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:58:55.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.295+0000 7f35aaa31640 1 -- 172.21.15.100:0/1142489496 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35a40a5310 con 0x7f35a40a15a0 2024-09-13T11:58:55.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.295+0000 7f35a0ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:58:55.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.295+0000 7f35a0ff9640 1 -- 172.21.15.100:0/1142489496 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f359803d680 con 0x7f35a40a15a0 2024-09-13T11:58:55.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.295+0000 7f35a0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:58:55.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.295+0000 7f35a0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:58:55.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.295+0000 7f35a0ff9640 10 monclient: dump: 2024-09-13T11:58:55.313 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:58:55.313 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:58:55.313 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:58:55.313 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:58:55.314 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:58:55.314 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:58:55.314 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:58:55.314 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:58:55.314 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:58:55.314 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:58:55.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.295+0000 7f35aaa31640 10 monclient: _renew_subs 2024-09-13T11:58:55.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.295+0000 7f35a27fc640 4 set_mon_vals no callback set 2024-09-13T11:58:55.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.295+0000 7f35aaa31640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:58:55.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.295+0000 7f35aaa31640 1 -- 172.21.15.100:0/1142489496 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35a41272a0 con 0x7f35a40a15a0 2024-09-13T11:58:55.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.295+0000 7f35a0ff9640 1 -- 172.21.15.100:0/1142489496 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f359803d820 con 0x7f35a40a15a0 2024-09-13T11:58:55.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.296+0000 7f35aaa31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:58:55.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.296+0000 7f35aaa31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:58:55.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.296+0000 7f35aaa31640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:58:55.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.296+0000 7f35aaa31640 1 -- 172.21.15.100:0/1142489496 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35a40a1980 con 0x7f35a40a15a0 2024-09-13T11:58:55.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.296+0000 7f35a0ff9640 1 --2- 172.21.15.100:0/1142489496 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f359003dfa0 0x7f3590040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:58:55.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.296+0000 7f35a0ff9640 1 -- 172.21.15.100:0/1142489496 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7f3598072bc0 con 0x7f35a40a15a0 2024-09-13T11:58:55.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.296+0000 7f35a3fff640 1 --2- 172.21.15.100:0/1142489496 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f359003dfa0 0x7f3590040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:58:55.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.296+0000 7f35a3fff640 10 monclient: get_auth_request con 0x7f359003dfa0 auth_method 0 2024-09-13T11:58:55.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.296+0000 7f35a3fff640 1 --2- 172.21.15.100:0/1142489496 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f359003dfa0 0x7f3590040460 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3594004f60 tx=0x7f3594007e30 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:58:55.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.300+0000 7f35a0ff9640 1 -- 172.21.15.100:0/1142489496 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f359803c3a0 con 0x7f35a40a15a0 2024-09-13T11:58:55.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.300+0000 7f35a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:58:55.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.300+0000 7f35a0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:58:55.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.540+0000 7f35aaa31640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:58:55.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.540+0000 7f35aaa31640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:58:55.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.540+0000 7f35aaa31640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:58:55.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.540+0000 7f35aaa31640 1 -- 172.21.15.100:0/1142489496 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f35a4127550 con 0x7f35a40a15a0 2024-09-13T11:58:55.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.540+0000 7f35a0ff9640 1 -- 172.21.15.100:0/1142489496 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f3598047460 con 0x7f35a40a15a0 2024-09-13T11:58:55.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.540+0000 7f35a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:58:55.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.540+0000 7f35a0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:58:55.543 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":26,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":96,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:58:55.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.542+0000 7f35aaa31640 1 -- 172.21.15.100:0/1142489496 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f359003dfa0 msgr2=0x7f3590040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:58:55.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.542+0000 7f35aaa31640 1 --2- 172.21.15.100:0/1142489496 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f359003dfa0 0x7f3590040460 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3594004f60 tx=0x7f3594007e30 comp rx=0 tx=0).stop 2024-09-13T11:58:55.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.542+0000 7f35aaa31640 10 monclient: shutdown 2024-09-13T11:58:55.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.542+0000 7f35aaa31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:58:55.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.542+0000 7f35aaa31640 1 -- 172.21.15.100:0/1142489496 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f35a40a15a0 msgr2=0x7f35a41ccc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:58:55.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.542+0000 7f35aaa31640 1 --2- 172.21.15.100:0/1142489496 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f35a40a15a0 0x7f35a41ccc40 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3598000c00 tx=0x7f35980046c0 comp rx=0 tx=0).stop 2024-09-13T11:58:55.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.542+0000 7f35aaa31640 1 -- 172.21.15.100:0/1142489496 shutdown_connections 2024-09-13T11:58:55.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.542+0000 7f35aaa31640 1 --2- 172.21.15.100:0/1142489496 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f359003dfa0 0x7f3590040460 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:58:55.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.542+0000 7f35aaa31640 1 --2- 172.21.15.100:0/1142489496 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f35a40a15a0 0x7f35a41ccc40 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:58:55.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.542+0000 7f35aaa31640 1 --2- 172.21.15.100:0/1142489496 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a40a2090 0x7f35a4124f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:58:55.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.542+0000 7f35aaa31640 1 --2- 172.21.15.100:0/1142489496 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f35a409f6d0 0x7f35a41cc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:58:55.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.542+0000 7f35aaa31640 1 -- 172.21.15.100:0/1142489496 >> 172.21.15.100:0/1142489496 conn(0x7f35a40a6470 msgr2=0x7f35a40717f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:58:55.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.542+0000 7f35aaa31640 1 -- 172.21.15.100:0/1142489496 shutdown_connections 2024-09-13T11:58:55.546 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:55.542+0000 7f35aaa31640 1 -- 172.21.15.100:0/1142489496 wait complete. 2024-09-13T11:58:58.554 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:58:58.662 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.661+0000 7f756a5f8640 10 monclient: get_monmap_and_config 2024-09-13T11:58:58.662 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.661+0000 7f756a5f8640 10 monclient: build_initial_monmap 2024-09-13T11:58:58.662 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.661+0000 7f756a5f8640 10 monclient: monmap: 2024-09-13T11:58:58.662 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:58:58.663 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:58:58.663 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:58:58.661959+0000 2024-09-13T11:58:58.663 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:58:58.661959+0000 2024-09-13T11:58:58.663 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:58:58.663 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:58:58.663 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:58:58.664 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:58:58.664 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:58:58.664 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:58:58.664 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:58:58.664 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:58:58.671 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.661+0000 7f756a5f8640 1 Processor -- start 2024-09-13T11:58:58.671 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.661+0000 7f756a5f8640 1 -- start start 2024-09-13T11:58:58.671 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.661+0000 7f756a5f8640 10 monclient: init 2024-09-13T11:58:58.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f756a5f8640 10 monclient: _reopen_session rank -1 2024-09-13T11:58:58.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f756a5f8640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-13T11:58:58.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f756a5f8640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7564137a60 0x7f75640a2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:58:58.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f756a5f8640 10 monclient(hunting): picked mon.noname-a con 0x7f7564137a60 addr v2:172.21.15.100:3300/0 2024-09-13T11:58:58.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f756a5f8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7564137e40 addr v1:172.21.15.100:6789/0 2024-09-13T11:58:58.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f756a5f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7564138250 addr v1:172.21.15.122:6789/0 2024-09-13T11:58:58.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f756a5f8640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75640a5770 con 0x7f7564137e40 2024-09-13T11:58:58.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f756a5f8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75640a5be0 con 0x7f7564138250 2024-09-13T11:58:58.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f756a5f8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:58:58.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f756a5f8640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f75640a4410 con 0x7f7564137a60 2024-09-13T11:58:58.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f756a5f8640 10 monclient(hunting): _renew_subs 2024-09-13T11:58:58.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f756a5f8640 10 monclient(hunting): authenticate will time out at 1119.255457s 2024-09-13T11:58:58.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f7563fff640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7564137a60 0x7f75640a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:58:58.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f75637fe640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f7564137e40 0x7f75640a4950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:43922/0 (socket says 172.21.15.100:43922) 2024-09-13T11:58:58.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f75637fe640 1 -- 172.21.15.100:0/3407645135 learned_addr learned my addr 172.21.15.100:0/3407645135 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:58:58.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f7563fff640 10 monclient(hunting): get_auth_request con 0x7f7564137a60 auth_method 0 2024-09-13T11:58:58.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f7563fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:58:58.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f7563fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:58:58.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f7563fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:58:58.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f7563fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:58:58.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f7563fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:58:58.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f7563fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:58:58.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f7563fff640 10 monclient(hunting): handle_auth_done global_id 15453 payload 274 2024-09-13T11:58:58.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f7563fff640 1 -- 172.21.15.100:0/3407645135 >> v1:172.21.15.122:6789/0 conn(0x7f7564138250 legacy=0x7f75640a5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:58:58.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f7563fff640 1 -- 172.21.15.100:0/3407645135 >> v1:172.21.15.100:6789/0 conn(0x7f7564137e40 legacy=0x7f75640a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:58:58.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f7563fff640 10 monclient: _finish_hunting 0 2024-09-13T11:58:58.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f7563fff640 1 monclient: found mon.noname-a 2024-09-13T11:58:58.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f7563fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:58:58.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f7563fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:58:58.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.662+0000 7f7563fff640 1 -- 172.21.15.100:0/3407645135 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75640a76b0 con 0x7f7564137a60 2024-09-13T11:58:58.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.663+0000 7f7563fff640 1 --2- 172.21.15.100:0/3407645135 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7564137a60 0x7f75640a2980 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f75640a5db0 tx=0x7f755402a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4090fce65b54154f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:58:58.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.663+0000 7f7562ffd640 1 -- 172.21.15.100:0/3407645135 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7554037070 con 0x7f7564137a60 2024-09-13T11:58:58.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.663+0000 7f7562ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:58:58.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.663+0000 7f7562ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:58:58.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.663+0000 7f7562ffd640 10 monclient: dump: 2024-09-13T11:58:58.674 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:58:58.674 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:58:58.674 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:58:58.674 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:58:58.675 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:58:58.675 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:58:58.675 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:58:58.675 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:58:58.675 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:58:58.675 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:58:58.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.663+0000 7f7562ffd640 10 monclient: _finish_auth 0 2024-09-13T11:58:58.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.663+0000 7f7562ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:58:58.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.663+0000 7f7562ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:58:58.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.663+0000 7f7562ffd640 1 -- 172.21.15.100:0/3407645135 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f755402adb0 con 0x7f7564137a60 2024-09-13T11:58:58.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.663+0000 7f756a5f8640 5 monclient: authenticate success, global_id 15453 2024-09-13T11:58:58.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.663+0000 7f756a5f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:58:58.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.663+0000 7f7562ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:58:58.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.663+0000 7f7562ffd640 1 -- 172.21.15.100:0/3407645135 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7554032660 con 0x7f7564137a60 2024-09-13T11:58:58.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.663+0000 7f7562ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:58:58.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.663+0000 7f7562ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:58:58.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.663+0000 7f7562ffd640 10 monclient: dump: 2024-09-13T11:58:58.676 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:58:58.676 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:58:58.676 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:58:58.676 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:58:58.676 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:58:58.676 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:58:58.676 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:58:58.676 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:58:58.676 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:58:58.676 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:58:58.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.663+0000 7f756a5f8640 10 monclient: get_monmap_and_config success 2024-09-13T11:58:58.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.663+0000 7f756a5f8640 4 set_mon_vals no callback set 2024-09-13T11:58:58.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.664+0000 7f756a5f8640 10 monclient: shutdown 2024-09-13T11:58:58.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.664+0000 7f756a5f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:58:58.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.664+0000 7f756a5f8640 1 -- 172.21.15.100:0/3407645135 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7564137a60 msgr2=0x7f75640a2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:58:58.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.664+0000 7f756a5f8640 1 --2- 172.21.15.100:0/3407645135 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7564137a60 0x7f75640a2980 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f75640a5db0 tx=0x7f755402a060 comp rx=0 tx=0).stop 2024-09-13T11:58:58.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.664+0000 7f756a5f8640 1 -- 172.21.15.100:0/3407645135 shutdown_connections 2024-09-13T11:58:58.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.664+0000 7f756a5f8640 1 --2- 172.21.15.100:0/3407645135 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7564137a60 0x7f75640a2980 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:58:58.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.664+0000 7f756a5f8640 1 -- 172.21.15.100:0/3407645135 >> 172.21.15.100:0/3407645135 conn(0x7f75640aa580 msgr2=0x7f75640a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:58:58.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.664+0000 7f756a5f8640 1 -- 172.21.15.100:0/3407645135 shutdown_connections 2024-09-13T11:58:58.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.664+0000 7f756a5f8640 1 -- 172.21.15.100:0/3407645135 wait complete. 2024-09-13T11:58:58.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.664+0000 7f756a5f8640 10 monclient: build_initial_monmap 2024-09-13T11:58:58.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.664+0000 7f756a5f8640 10 monclient: monmap: 2024-09-13T11:58:58.677 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:58:58.678 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:58:58.678 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:58:58.678 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:58:58.678 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:58:58.678 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:58:58.678 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:58:58.678 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:58:58.678 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:58:58.678 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:58:58.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.665+0000 7f756a5f8640 1 Processor -- start 2024-09-13T11:58:58.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.665+0000 7f756a5f8640 1 -- start start 2024-09-13T11:58:58.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.665+0000 7f756a5f8640 10 monclient: init 2024-09-13T11:58:58.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.665+0000 7f756a5f8640 10 monclient: _reopen_session rank -1 2024-09-13T11:58:58.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.665+0000 7f756a5f8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:58:58.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.665+0000 7f756a5f8640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7564137a60 0x7f756413eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:58:58.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.665+0000 7f756a5f8640 10 monclient(hunting): picked mon.noname-c con 0x7f7564137a60 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:58:58.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.665+0000 7f756a5f8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7564137e40 0x7f75641cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:58:58.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f756a5f8640 10 monclient(hunting): picked mon.noname-b con 0x7f7564137e40 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:58:58.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f756a5f8640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7564138250 0x7f756413c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:58:58.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f756a5f8640 10 monclient(hunting): picked mon.noname-a con 0x7f7564138250 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:58:58.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f756a5f8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:58:58.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f756a5f8640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f756406fd80 con 0x7f7564138250 2024-09-13T11:58:58.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f756a5f8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:58:58.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f756a5f8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f75640a1700 con 0x7f7564137e40 2024-09-13T11:58:58.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f756a5f8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:58:58.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f756a5f8640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f756406ff00 con 0x7f7564137a60 2024-09-13T11:58:58.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f756a5f8640 10 monclient(hunting): _renew_subs 2024-09-13T11:58:58.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f756a5f8640 10 monclient(hunting): authenticate will time out at 1119.259052s 2024-09-13T11:58:58.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7563fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7564137a60 0x7f756413eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:58:58.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7568b95640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7564138250 0x7f756413c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:58:58.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7563fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7564137a60 0x7f756413eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:51964/0 (socket says 172.21.15.100:51964) 2024-09-13T11:58:58.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7563fff640 1 -- 172.21.15.100:0/2342364859 learned_addr learned my addr 172.21.15.100:0/2342364859 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:58:58.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7563fff640 10 monclient(hunting): get_auth_request con 0x7f7564137a60 auth_method 0 2024-09-13T11:58:58.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7563fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:58:58.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7563fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:58:58.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7563fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:58:58.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7568b95640 10 monclient(hunting): get_auth_request con 0x7f7564138250 auth_method 0 2024-09-13T11:58:58.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7568b95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:58:58.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7568b95640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:58:58.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7568b95640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:58:58.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7568b95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:58:58.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7568b95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:58:58.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7568b95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:58:58.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7563fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:58:58.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7563fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:58:58.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7563fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:58:58.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f75637fe640 1 --2- 172.21.15.100:0/2342364859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7564137e40 0x7f75641cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:58:58.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7568b95640 10 monclient(hunting): handle_auth_done global_id 15459 payload 824 2024-09-13T11:58:58.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7568b95640 1 -- 172.21.15.100:0/2342364859 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7564137a60 msgr2=0x7f756413eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:58:58.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7568b95640 1 --2- 172.21.15.100:0/2342364859 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7564137a60 0x7f756413eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:58:58.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7568b95640 1 -- 172.21.15.100:0/2342364859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7564137e40 msgr2=0x7f75641cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:58:58.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7568b95640 1 --2- 172.21.15.100:0/2342364859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7564137e40 0x7f75641cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:58:58.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7568b95640 10 monclient: _finish_hunting 0 2024-09-13T11:58:58.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7568b95640 1 monclient: found mon.noname-a 2024-09-13T11:58:58.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7568b95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:58:58.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7568b95640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:58:58.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7568b95640 1 -- 172.21.15.100:0/2342364859 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75640a1140 con 0x7f7564138250 2024-09-13T11:58:58.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f75637fe640 10 monclient: get_auth_request con 0x7f7564137e40 auth_method 0 2024-09-13T11:58:58.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f75637fe640 1 --2- 172.21.15.100:0/2342364859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7564137e40 0x7f75641cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:58:58.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7563fff640 1 --2- 172.21.15.100:0/2342364859 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7564137a60 0x7f756413eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:58:58.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7568b95640 1 --2- 172.21.15.100:0/2342364859 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7564138250 0x7f756413c4e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f755c00c780 tx=0x7f755c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:58:58.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7560ff9640 1 -- 172.21.15.100:0/2342364859 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f755c016020 con 0x7f7564138250 2024-09-13T11:58:58.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7560ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:58:58.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7560ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:58:58.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7560ff9640 10 monclient: dump: 2024-09-13T11:58:58.683 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:58:58.683 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:58:58.683 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:58:58.683 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:58:58.683 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:58:58.683 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:58:58.683 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:58:58.683 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:58:58.683 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:58:58.683 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:58:58.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7560ff9640 10 monclient: _finish_auth 0 2024-09-13T11:58:58.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7560ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:58:58.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7560ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:58:58.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f756a5f8640 5 monclient: authenticate success, global_id 15459 2024-09-13T11:58:58.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7560ff9640 1 -- 172.21.15.100:0/2342364859 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f755c00aa50 con 0x7f7564138250 2024-09-13T11:58:58.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7560ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:58:58.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f756a5f8640 10 monclient: _renew_subs 2024-09-13T11:58:58.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7560ff9640 1 -- 172.21.15.100:0/2342364859 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f755c00ce30 con 0x7f7564138250 2024-09-13T11:58:58.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f756a5f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:58:58.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f756a5f8640 1 -- 172.21.15.100:0/2342364859 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7564136d30 con 0x7f7564138250 2024-09-13T11:58:58.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7560ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:58:58.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7560ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:58:58.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f7560ff9640 10 monclient: dump: 2024-09-13T11:58:58.684 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:58:58.684 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:58:58.684 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:58:58.685 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:58:58.685 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:58:58.685 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:58:58.685 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:58:58.685 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:58:58.685 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:58:58.685 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:58:58.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f75627fc640 4 set_mon_vals no callback set 2024-09-13T11:58:58.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f756a5f8640 10 monclient: _renew_subs 2024-09-13T11:58:58.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f756a5f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:58:58.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.666+0000 7f756a5f8640 1 -- 172.21.15.100:0/2342364859 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7564140050 con 0x7f7564138250 2024-09-13T11:58:58.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.667+0000 7f7560ff9640 1 -- 172.21.15.100:0/2342364859 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f755c019020 con 0x7f7564138250 2024-09-13T11:58:58.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.667+0000 7f756a5f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:58:58.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.667+0000 7f7560ff9640 1 --2- 172.21.15.100:0/2342364859 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f755003dfa0 0x7f7550040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:58:58.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.667+0000 7f756a5f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:58:58.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.667+0000 7f756a5f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:58:58.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.667+0000 7f756a5f8640 1 -- 172.21.15.100:0/2342364859 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75641c58e0 con 0x7f7564138250 2024-09-13T11:58:58.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.668+0000 7f7560ff9640 1 -- 172.21.15.100:0/2342364859 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7f755c052410 con 0x7f7564138250 2024-09-13T11:58:58.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.668+0000 7f7563fff640 1 --2- 172.21.15.100:0/2342364859 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f755003dfa0 0x7f7550040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:58:58.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.668+0000 7f7563fff640 10 monclient: get_auth_request con 0x7f755003dfa0 auth_method 0 2024-09-13T11:58:58.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.668+0000 7f7563fff640 1 --2- 172.21.15.100:0/2342364859 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f755003dfa0 0x7f7550040460 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7554009560 tx=0x7f755402e000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:58:58.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.672+0000 7f7560ff9640 1 -- 172.21.15.100:0/2342364859 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f755c039e20 con 0x7f7564138250 2024-09-13T11:58:58.687 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.672+0000 7f7560ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:58:58.687 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.672+0000 7f7560ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:58:58.917 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.913+0000 7f756a5f8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:58:58.917 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.913+0000 7f756a5f8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:58:58.917 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.913+0000 7f756a5f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:58:58.917 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.913+0000 7f756a5f8640 1 -- 172.21.15.100:0/2342364859 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f75640a0670 con 0x7f7564138250 2024-09-13T11:58:58.917 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.914+0000 7f7560ff9640 1 -- 172.21.15.100:0/2342364859 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f755c020030 con 0x7f7564138250 2024-09-13T11:58:58.917 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.914+0000 7f7560ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:58:58.917 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.914+0000 7f7560ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:58:58.918 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":26,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":99,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:58:58.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.916+0000 7f756a5f8640 1 -- 172.21.15.100:0/2342364859 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f755003dfa0 msgr2=0x7f7550040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:58:58.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.916+0000 7f756a5f8640 1 --2- 172.21.15.100:0/2342364859 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f755003dfa0 0x7f7550040460 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7554009560 tx=0x7f755402e000 comp rx=0 tx=0).stop 2024-09-13T11:58:58.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.916+0000 7f756a5f8640 10 monclient: shutdown 2024-09-13T11:58:58.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.916+0000 7f756a5f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:58:58.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.916+0000 7f756a5f8640 1 -- 172.21.15.100:0/2342364859 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7564138250 msgr2=0x7f756413c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:58:58.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.916+0000 7f756a5f8640 1 --2- 172.21.15.100:0/2342364859 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7564138250 0x7f756413c4e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f755c00c780 tx=0x7f755c00cc40 comp rx=0 tx=0).stop 2024-09-13T11:58:58.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.916+0000 7f756a5f8640 1 -- 172.21.15.100:0/2342364859 shutdown_connections 2024-09-13T11:58:58.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.916+0000 7f756a5f8640 1 --2- 172.21.15.100:0/2342364859 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f755003dfa0 0x7f7550040460 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:58:58.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.916+0000 7f756a5f8640 1 --2- 172.21.15.100:0/2342364859 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7564138250 0x7f756413c4e0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:58:58.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.916+0000 7f756a5f8640 1 --2- 172.21.15.100:0/2342364859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7564137e40 0x7f75641cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:58:58.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.916+0000 7f756a5f8640 1 --2- 172.21.15.100:0/2342364859 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7564137a60 0x7f756413eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:58:58.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.916+0000 7f756a5f8640 1 -- 172.21.15.100:0/2342364859 >> 172.21.15.100:0/2342364859 conn(0x7f75640aa580 msgr2=0x7f75640a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:58:58.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.916+0000 7f756a5f8640 1 -- 172.21.15.100:0/2342364859 shutdown_connections 2024-09-13T11:58:58.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:58:58.916+0000 7f756a5f8640 1 -- 172.21.15.100:0/2342364859 wait complete. 2024-09-13T11:59:01.929 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:59:02.037 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.035+0000 7f081e189640 10 monclient: get_monmap_and_config 2024-09-13T11:59:02.037 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.035+0000 7f081e189640 10 monclient: build_initial_monmap 2024-09-13T11:59:02.037 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.035+0000 7f081e189640 10 monclient: monmap: 2024-09-13T11:59:02.037 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:02.037 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.037 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:02.036699+0000 2024-09-13T11:59:02.037 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:02.036699+0000 2024-09-13T11:59:02.037 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:02.037 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.037 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:02.038 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:02.038 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:02.038 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:02.038 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:02.038 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.044 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.036+0000 7f081e189640 1 Processor -- start 2024-09-13T11:59:02.044 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.036+0000 7f081e189640 1 -- start start 2024-09-13T11:59:02.044 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.036+0000 7f081e189640 10 monclient: init 2024-09-13T11:59:02.044 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f081e189640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:02.044 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f081e189640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-13T11:59:02.044 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f081e189640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f08180a37d0 addr v1:172.21.15.122:6789/0 2024-09-13T11:59:02.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f081e189640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f081806eeb0 0x7f081813cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f081e189640 10 monclient(hunting): picked mon.noname-c con 0x7f081806eeb0 addr v2:172.21.15.122:3300/0 2024-09-13T11:59:02.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f081e189640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f08180a42c0 0x7f081813f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f081e189640 10 monclient(hunting): picked mon.noname-a con 0x7f08180a42c0 addr v2:172.21.15.100:3300/0 2024-09-13T11:59:02.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f081e189640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f081806e950 con 0x7f08180a37d0 2024-09-13T11:59:02.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f081e189640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f081e189640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f081809fde0 con 0x7f08180a42c0 2024-09-13T11:59:02.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f081e189640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f081e189640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f081806f330 con 0x7f081806eeb0 2024-09-13T11:59:02.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f081e189640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:02.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f081e189640 10 monclient(hunting): authenticate will time out at 1122.630451s 2024-09-13T11:59:02.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f0817fff640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f08180a42c0 0x7f081813f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f0817fff640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f08180a42c0 0x7f081813f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:60044/0 (socket says 172.21.15.100:60044) 2024-09-13T11:59:02.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f0817fff640 1 -- 172.21.15.100:0/1372296261 learned_addr learned my addr 172.21.15.100:0/1372296261 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:02.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f0817fff640 10 monclient(hunting): get_auth_request con 0x7f08180a42c0 auth_method 0 2024-09-13T11:59:02.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f0817fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f0817fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f0817fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f0817fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f0817fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f0817fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:02.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f080ffff640 1 --2- 172.21.15.100:0/1372296261 >> v2:172.21.15.122:3300/0 conn(0x7f081806eeb0 0x7f081813cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f0817fff640 10 monclient(hunting): handle_auth_done global_id 15462 payload 274 2024-09-13T11:59:02.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f0817fff640 1 -- 172.21.15.100:0/1372296261 >> v2:172.21.15.122:3300/0 conn(0x7f081806eeb0 msgr2=0x7f081813cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:02.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f0817fff640 1 --2- 172.21.15.100:0/1372296261 >> v2:172.21.15.122:3300/0 conn(0x7f081806eeb0 0x7f081813cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f0817fff640 1 -- 172.21.15.100:0/1372296261 >> v1:172.21.15.122:6789/0 conn(0x7f08180a37d0 legacy=0x7f08180a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:02.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f0817fff640 10 monclient: _finish_hunting 0 2024-09-13T11:59:02.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f0817fff640 1 monclient: found mon.noname-a 2024-09-13T11:59:02.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f0817fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:02.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f0817fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:02.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f0817fff640 1 -- 172.21.15.100:0/1372296261 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08180a5240 con 0x7f08180a42c0 2024-09-13T11:59:02.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f080ffff640 10 monclient: get_auth_request con 0x7f081806eeb0 auth_method 0 2024-09-13T11:59:02.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.037+0000 7f080ffff640 1 --2- 172.21.15.100:0/1372296261 >> v2:172.21.15.122:3300/0 conn(0x7f081806eeb0 0x7f081813cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:59:02.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.038+0000 7f0817fff640 1 --2- 172.21.15.100:0/1372296261 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08180a42c0 0x7f081813f170 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f081806f540 tx=0x7f081002a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a218a93af9bccf84 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:02.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.038+0000 7f0816ffd640 1 -- 172.21.15.100:0/1372296261 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0810036070 con 0x7f08180a42c0 2024-09-13T11:59:02.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.038+0000 7f0816ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.048 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.038+0000 7f0816ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:02.048 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.038+0000 7f0816ffd640 10 monclient: dump: 2024-09-13T11:59:02.048 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.048 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.048 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.048 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.048 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.048 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.048 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.048 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.048 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.048 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.048 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.038+0000 7f0816ffd640 10 monclient: _finish_auth 0 2024-09-13T11:59:02.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.038+0000 7f0816ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:59:02.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.038+0000 7f0816ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:02.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.038+0000 7f081e189640 5 monclient: authenticate success, global_id 15462 2024-09-13T11:59:02.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.038+0000 7f081e189640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:02.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.038+0000 7f0816ffd640 1 -- 172.21.15.100:0/1372296261 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0810031600 con 0x7f08180a42c0 2024-09-13T11:59:02.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.038+0000 7f0816ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:02.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.038+0000 7f081e189640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:02.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.038+0000 7f081e189640 4 set_mon_vals no callback set 2024-09-13T11:59:02.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.038+0000 7f0816ffd640 1 -- 172.21.15.100:0/1372296261 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08100319d0 con 0x7f08180a42c0 2024-09-13T11:59:02.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.038+0000 7f0816ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.038+0000 7f0816ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:02.049 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.038+0000 7f0816ffd640 10 monclient: dump: 2024-09-13T11:59:02.049 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.050 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.050 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.050 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.050 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.050 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.050 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.050 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.050 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.050 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.039+0000 7f081e189640 10 monclient: shutdown 2024-09-13T11:59:02.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.039+0000 7f081e189640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:02.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.039+0000 7f081e189640 1 -- 172.21.15.100:0/1372296261 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08180a42c0 msgr2=0x7f081813f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.039+0000 7f081e189640 1 --2- 172.21.15.100:0/1372296261 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08180a42c0 0x7f081813f170 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f081806f540 tx=0x7f081002a570 comp rx=0 tx=0).stop 2024-09-13T11:59:02.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.039+0000 7f081e189640 1 -- 172.21.15.100:0/1372296261 shutdown_connections 2024-09-13T11:59:02.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.039+0000 7f081e189640 1 --2- 172.21.15.100:0/1372296261 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08180a42c0 0x7f081813f170 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.039+0000 7f081e189640 1 --2- 172.21.15.100:0/1372296261 >> v2:172.21.15.122:3300/0 conn(0x7f081806eeb0 0x7f081813cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.039+0000 7f081e189640 1 -- 172.21.15.100:0/1372296261 >> 172.21.15.100:0/1372296261 conn(0x7f08180a83f0 msgr2=0x7f08180a48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:02.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.039+0000 7f081e189640 1 -- 172.21.15.100:0/1372296261 shutdown_connections 2024-09-13T11:59:02.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.039+0000 7f081e189640 1 -- 172.21.15.100:0/1372296261 wait complete. 2024-09-13T11:59:02.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.039+0000 7f081e189640 10 monclient: build_initial_monmap 2024-09-13T11:59:02.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.039+0000 7f081e189640 10 monclient: monmap: 2024-09-13T11:59:02.051 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:02.051 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:02.051 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:02.051 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:02.051 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:02.052 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.052 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:02.052 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:02.052 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:02.052 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f081e189640 1 Processor -- start 2024-09-13T11:59:02.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f081e189640 1 -- start start 2024-09-13T11:59:02.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f081e189640 10 monclient: init 2024-09-13T11:59:02.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f081e189640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:02.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f081e189640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:59:02.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f081e189640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f081806eeb0 0x7f081813a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f081e189640 10 monclient(hunting): picked mon.noname-c con 0x7f081806eeb0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:02.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f081e189640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08180a37d0 0x7f081813adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f081e189640 10 monclient(hunting): picked mon.noname-a con 0x7f08180a37d0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:02.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f081e189640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08180a42c0 0x7f081813b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f081e189640 10 monclient(hunting): picked mon.noname-b con 0x7f08180a42c0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:02.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f081e189640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f081e189640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f081813f830 con 0x7f08180a37d0 2024-09-13T11:59:02.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f081e189640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f081e189640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f081813f6b0 con 0x7f08180a42c0 2024-09-13T11:59:02.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f081e189640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f081e189640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f081813f9b0 con 0x7f081806eeb0 2024-09-13T11:59:02.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f081e189640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:02.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f081e189640 10 monclient(hunting): authenticate will time out at 1122.633821s 2024-09-13T11:59:02.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f080ffff640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08180a37d0 0x7f081813adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f080ffff640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08180a37d0 0x7f081813adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:60058/0 (socket says 172.21.15.100:60058) 2024-09-13T11:59:02.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f080ffff640 1 -- 172.21.15.100:0/2422975450 learned_addr learned my addr 172.21.15.100:0/2422975450 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:02.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f08177fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f081806eeb0 0x7f081813a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f080ffff640 10 monclient(hunting): get_auth_request con 0x7f08180a37d0 auth_method 0 2024-09-13T11:59:02.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f080ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f080ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f080ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f08177fe640 10 monclient(hunting): get_auth_request con 0x7f081806eeb0 auth_method 0 2024-09-13T11:59:02.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f08177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f08177fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.040+0000 7f08177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f080ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f080ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.055 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f080ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:02.055 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f08177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.055 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f08177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.055 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f08177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:02.055 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f0817fff640 1 --2- 172.21.15.100:0/2422975450 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08180a42c0 0x7f081813b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.055 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f0817fff640 10 monclient(hunting): get_auth_request con 0x7f08180a42c0 auth_method 0 2024-09-13T11:59:02.055 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f0817fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.055 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f0817fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.055 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f0817fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.055 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f080ffff640 10 monclient(hunting): handle_auth_done global_id 15465 payload 824 2024-09-13T11:59:02.055 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f080ffff640 1 -- 172.21.15.100:0/2422975450 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f081806eeb0 msgr2=0x7f081813a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.055 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f080ffff640 1 --2- 172.21.15.100:0/2422975450 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f081806eeb0 0x7f081813a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.055 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f080ffff640 1 -- 172.21.15.100:0/2422975450 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08180a42c0 msgr2=0x7f081813b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.056 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f080ffff640 1 --2- 172.21.15.100:0/2422975450 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08180a42c0 0x7f081813b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.056 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f080ffff640 10 monclient: _finish_hunting 0 2024-09-13T11:59:02.056 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f080ffff640 1 monclient: found mon.noname-a 2024-09-13T11:59:02.056 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f080ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:02.056 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f080ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:02.056 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f080ffff640 1 -- 172.21.15.100:0/2422975450 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f081813d6a0 con 0x7f08180a37d0 2024-09-13T11:59:02.056 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f0817fff640 1 --2- 172.21.15.100:0/2422975450 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08180a42c0 0x7f081813b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:59:02.056 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f080ffff640 1 --2- 172.21.15.100:0/2422975450 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08180a37d0 0x7f081813adc0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f080000c7f0 tx=0x7f080000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:02.056 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f0814ff9640 1 -- 172.21.15.100:0/2422975450 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0800015070 con 0x7f08180a37d0 2024-09-13T11:59:02.056 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f0814ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.056 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f0814ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:02.057 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f0814ff9640 10 monclient: dump: 2024-09-13T11:59:02.057 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.057 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.057 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.057 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.057 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.057 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.057 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.057 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.057 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.057 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.057 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f0814ff9640 10 monclient: _finish_auth 0 2024-09-13T11:59:02.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f0814ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:59:02.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f0814ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:02.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f0814ff9640 1 -- 172.21.15.100:0/2422975450 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0800007820 con 0x7f08180a37d0 2024-09-13T11:59:02.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f081e189640 5 monclient: authenticate success, global_id 15465 2024-09-13T11:59:02.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f081e189640 10 monclient: _renew_subs 2024-09-13T11:59:02.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f081e189640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:02.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f081e189640 1 -- 172.21.15.100:0/2422975450 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f081813e510 con 0x7f08180a37d0 2024-09-13T11:59:02.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f0814ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:02.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f0814ff9640 1 -- 172.21.15.100:0/2422975450 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0800007bd0 con 0x7f08180a37d0 2024-09-13T11:59:02.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f0814ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f0814ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:02.058 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f0814ff9640 10 monclient: dump: 2024-09-13T11:59:02.059 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.059 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.059 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.059 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.059 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.059 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.059 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.059 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.059 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.059 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.059 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f08167fc640 4 set_mon_vals no callback set 2024-09-13T11:59:02.060 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f081e189640 10 monclient: _renew_subs 2024-09-13T11:59:02.060 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f081e189640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:02.060 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.041+0000 7f081e189640 1 -- 172.21.15.100:0/2422975450 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f081813e230 con 0x7f08180a37d0 2024-09-13T11:59:02.060 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.042+0000 7f0814ff9640 1 -- 172.21.15.100:0/2422975450 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f0800018070 con 0x7f08180a37d0 2024-09-13T11:59:02.060 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.042+0000 7f081e189640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:02.060 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.042+0000 7f081e189640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:02.060 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.042+0000 7f081e189640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:02.060 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.042+0000 7f081e189640 1 -- 172.21.15.100:0/2422975450 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08180a3bb0 con 0x7f08180a37d0 2024-09-13T11:59:02.060 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.042+0000 7f0814ff9640 1 --2- 172.21.15.100:0/2422975450 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f07f803dfa0 0x7f07f8040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.060 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.042+0000 7f0814ff9640 1 -- 172.21.15.100:0/2422975450 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7f0800052790 con 0x7f08180a37d0 2024-09-13T11:59:02.060 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.043+0000 7f08177fe640 1 --2- 172.21.15.100:0/2422975450 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f07f803dfa0 0x7f07f8040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.060 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.043+0000 7f08177fe640 10 monclient: get_auth_request con 0x7f07f803dfa0 auth_method 0 2024-09-13T11:59:02.060 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.043+0000 7f08177fe640 1 --2- 172.21.15.100:0/2422975450 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f07f803dfa0 0x7f07f8040460 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0808004f60 tx=0x7f0808007e30 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:02.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.047+0000 7f0814ff9640 1 -- 172.21.15.100:0/2422975450 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f0800054d00 con 0x7f08180a37d0 2024-09-13T11:59:02.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.047+0000 7f0814ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:02.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.047+0000 7f0814ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:59:02.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.290+0000 7f081e189640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:59:02.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.290+0000 7f081e189640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:59:02.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.290+0000 7f081e189640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:02.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.290+0000 7f081e189640 1 -- 172.21.15.100:0/2422975450 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f081806ca60 con 0x7f08180a37d0 2024-09-13T11:59:02.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.290+0000 7f0814ff9640 1 -- 172.21.15.100:0/2422975450 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f0800011ba0 con 0x7f08180a37d0 2024-09-13T11:59:02.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.290+0000 7f0814ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:59:02.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.290+0000 7f0814ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:59:02.292 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:59:02.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.292+0000 7f081e189640 1 -- 172.21.15.100:0/2422975450 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f07f803dfa0 msgr2=0x7f07f8040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.292+0000 7f081e189640 1 --2- 172.21.15.100:0/2422975450 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f07f803dfa0 0x7f07f8040460 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0808004f60 tx=0x7f0808007e30 comp rx=0 tx=0).stop 2024-09-13T11:59:02.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.292+0000 7f081e189640 10 monclient: shutdown 2024-09-13T11:59:02.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.292+0000 7f081e189640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:02.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.292+0000 7f081e189640 1 -- 172.21.15.100:0/2422975450 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08180a37d0 msgr2=0x7f081813adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.292+0000 7f081e189640 1 --2- 172.21.15.100:0/2422975450 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08180a37d0 0x7f081813adc0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f080000c7f0 tx=0x7f080000ccb0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.292+0000 7f081e189640 1 -- 172.21.15.100:0/2422975450 shutdown_connections 2024-09-13T11:59:02.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.292+0000 7f081e189640 1 --2- 172.21.15.100:0/2422975450 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f07f803dfa0 0x7f07f8040460 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.292+0000 7f081e189640 1 --2- 172.21.15.100:0/2422975450 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f08180a37d0 0x7f081813adc0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.292+0000 7f081e189640 1 --2- 172.21.15.100:0/2422975450 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08180a42c0 0x7f081813b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.295 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.292+0000 7f081e189640 1 --2- 172.21.15.100:0/2422975450 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f081806eeb0 0x7f081813a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.295 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.292+0000 7f081e189640 1 -- 172.21.15.100:0/2422975450 >> 172.21.15.100:0/2422975450 conn(0x7f08180a83f0 msgr2=0x7f081806f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:02.295 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.292+0000 7f081e189640 1 -- 172.21.15.100:0/2422975450 shutdown_connections 2024-09-13T11:59:02.295 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.292+0000 7f081e189640 1 -- 172.21.15.100:0/2422975450 wait complete. 2024-09-13T11:59:02.304 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-13T11:59:02.304 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-13T11:59:02.304 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-13T11:59:02.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.401+0000 7fe8e70a9640 10 monclient: get_monmap_and_config 2024-09-13T11:59:02.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.401+0000 7fe8e70a9640 10 monclient: build_initial_monmap 2024-09-13T11:59:02.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.401+0000 7fe8e70a9640 10 monclient: monmap: 2024-09-13T11:59:02.403 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:02.403 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.403 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:02.402509+0000 2024-09-13T11:59:02.403 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:02.402509+0000 2024-09-13T11:59:02.403 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:02.403 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.403 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:02.403 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:02.403 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:02.403 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:02.403 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:02.403 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.402+0000 7fe8e70a9640 1 Processor -- start 2024-09-13T11:59:02.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.402+0000 7fe8e70a9640 1 -- start start 2024-09-13T11:59:02.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.402+0000 7fe8e70a9640 10 monclient: init 2024-09-13T11:59:02.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.402+0000 7fe8e70a9640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:02.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.402+0000 7fe8e70a9640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-13T11:59:02.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.402+0000 7fe8e70a9640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8e00a15a0 0x7fe8e00a1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.402+0000 7fe8e70a9640 10 monclient(hunting): picked mon.noname-b con 0x7fe8e00a15a0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:02.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.402+0000 7fe8e70a9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe8e009f6d0 0x7fe8e00a1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.402+0000 7fe8e70a9640 10 monclient(hunting): picked mon.noname-c con 0x7fe8e009f6d0 addr v2:172.21.15.122:3300/0 2024-09-13T11:59:02.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.402+0000 7fe8e70a9640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe8e01343d0 0x7fe8e0134860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.414 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.402+0000 7fe8e70a9640 10 monclient(hunting): picked mon.noname-a con 0x7fe8e01343d0 addr v2:172.21.15.100:3300/0 2024-09-13T11:59:02.415 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.402+0000 7fe8e70a9640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.415 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.402+0000 7fe8e70a9640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7fe8e00a2400 con 0x7fe8e01343d0 2024-09-13T11:59:02.415 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.402+0000 7fe8e70a9640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.415 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.402+0000 7fe8e70a9640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7fe8e009d3b0 con 0x7fe8e009f6d0 2024-09-13T11:59:02.415 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.402+0000 7fe8e70a9640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.415 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.402+0000 7fe8e70a9640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fe8e009ecf0 con 0x7fe8e00a15a0 2024-09-13T11:59:02.415 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.402+0000 7fe8e70a9640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:02.415 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.402+0000 7fe8e70a9640 10 monclient(hunting): authenticate will time out at 1122.995902s 2024-09-13T11:59:02.415 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e4e45640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8e00a15a0 0x7fe8e00a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.415 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e5646640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe8e01343d0 0x7fe8e0134860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.415 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e5646640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe8e01343d0 0x7fe8e0134860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47028/0 (socket says 172.21.15.100:47028) 2024-09-13T11:59:02.415 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e4e45640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8e00a15a0 0x7fe8e00a1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:51786/0 (socket says 172.21.15.100:51786) 2024-09-13T11:59:02.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e5646640 1 -- 172.21.15.100:0/1002931948 learned_addr learned my addr 172.21.15.100:0/1002931948 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:02.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e5646640 10 monclient(hunting): get_auth_request con 0x7fe8e01343d0 auth_method 0 2024-09-13T11:59:02.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e5646640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e5646640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e5646640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e4e45640 10 monclient(hunting): get_auth_request con 0x7fe8e00a15a0 auth_method 0 2024-09-13T11:59:02.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e4e45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e4e45640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e4e45640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e4e45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e4e45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.416 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e4e45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:02.417 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e5646640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.417 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e5646640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.417 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e5646640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:02.417 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8dffff640 1 --2- 172.21.15.100:0/1002931948 >> v2:172.21.15.122:3300/0 conn(0x7fe8e009f6d0 0x7fe8e00a1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.417 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8dffff640 10 monclient(hunting): get_auth_request con 0x7fe8e009f6d0 auth_method 0 2024-09-13T11:59:02.417 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.417 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8dffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.417 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.417 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e5646640 10 monclient(hunting): handle_auth_done global_id 15468 payload 274 2024-09-13T11:59:02.417 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e5646640 1 -- 172.21.15.100:0/1002931948 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8e00a15a0 msgr2=0x7fe8e00a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:02.417 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e5646640 1 --2- 172.21.15.100:0/1002931948 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8e00a15a0 0x7fe8e00a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.417 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e5646640 1 -- 172.21.15.100:0/1002931948 >> v2:172.21.15.122:3300/0 conn(0x7fe8e009f6d0 msgr2=0x7fe8e00a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:02.418 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e5646640 1 --2- 172.21.15.100:0/1002931948 >> v2:172.21.15.122:3300/0 conn(0x7fe8e009f6d0 0x7fe8e00a1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.418 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e5646640 10 monclient: _finish_hunting 0 2024-09-13T11:59:02.418 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e5646640 1 monclient: found mon.noname-a 2024-09-13T11:59:02.418 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e5646640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:02.418 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e5646640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:02.418 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e5646640 1 -- 172.21.15.100:0/1002931948 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8e009ea10 con 0x7fe8e01343d0 2024-09-13T11:59:02.418 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8e4e45640 1 --2- 172.21.15.100:0/1002931948 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8e00a15a0 0x7fe8e00a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:59:02.418 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.403+0000 7fe8dffff640 1 --2- 172.21.15.100:0/1002931948 >> v2:172.21.15.122:3300/0 conn(0x7fe8e009f6d0 0x7fe8e00a1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:59:02.418 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8e5646640 1 --2- 172.21.15.100:0/1002931948 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e01343d0 0x7fe8e0134860 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fe8e00a3700 tx=0x7fe8d80093b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fb91093befca41a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:02.418 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8df7fe640 1 -- 172.21.15.100:0/1002931948 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d8036070 con 0x7fe8e01343d0 2024-09-13T11:59:02.418 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.418 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8df7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:02.418 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8df7fe640 10 monclient: dump: 2024-09-13T11:59:02.419 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.419 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.419 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.419 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.419 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.419 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.419 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.419 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.419 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.419 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8df7fe640 10 monclient: _finish_auth 0 2024-09-13T11:59:02.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8df7fe640 10 monclient: _check_auth_tickets 2024-09-13T11:59:02.419 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:02.420 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8e70a9640 5 monclient: authenticate success, global_id 15468 2024-09-13T11:59:02.420 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8df7fe640 1 -- 172.21.15.100:0/1002931948 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8d8009600 con 0x7fe8e01343d0 2024-09-13T11:59:02.420 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8e70a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:02.420 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8df7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:02.420 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8e70a9640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:02.420 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8e70a9640 4 set_mon_vals no callback set 2024-09-13T11:59:02.420 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8df7fe640 1 -- 172.21.15.100:0/1002931948 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d80099f0 con 0x7fe8e01343d0 2024-09-13T11:59:02.420 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.420 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:02.420 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8df7fe640 10 monclient: dump: 2024-09-13T11:59:02.421 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.421 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.421 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.421 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.421 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.421 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.421 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.421 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.422 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.422 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8e70a9640 10 monclient: shutdown 2024-09-13T11:59:02.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8e70a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:02.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8e70a9640 1 -- 172.21.15.100:0/1002931948 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e01343d0 msgr2=0x7fe8e0134860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8e70a9640 1 --2- 172.21.15.100:0/1002931948 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e01343d0 0x7fe8e0134860 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fe8e00a3700 tx=0x7fe8d80093b0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8e70a9640 1 -- 172.21.15.100:0/1002931948 shutdown_connections 2024-09-13T11:59:02.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8e70a9640 1 --2- 172.21.15.100:0/1002931948 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e01343d0 0x7fe8e0134860 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8e70a9640 1 --2- 172.21.15.100:0/1002931948 >> v2:172.21.15.122:3300/0 conn(0x7fe8e009f6d0 0x7fe8e00a1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8e70a9640 1 --2- 172.21.15.100:0/1002931948 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8e00a15a0 0x7fe8e00a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8e70a9640 1 -- 172.21.15.100:0/1002931948 >> 172.21.15.100:0/1002931948 conn(0x7fe8e00a6470 msgr2=0x7fe8e00a2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:02.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8e70a9640 1 -- 172.21.15.100:0/1002931948 shutdown_connections 2024-09-13T11:59:02.422 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.404+0000 7fe8e70a9640 1 -- 172.21.15.100:0/1002931948 wait complete. 2024-09-13T11:59:02.423 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.405+0000 7fe8e70a9640 10 monclient: build_initial_monmap 2024-09-13T11:59:02.423 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.405+0000 7fe8e70a9640 10 monclient: monmap: 2024-09-13T11:59:02.423 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:02.423 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:02.423 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:02.423 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:02.423 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:02.423 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.423 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:02.423 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:02.423 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:02.423 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.423 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.405+0000 7fe8e70a9640 1 Processor -- start 2024-09-13T11:59:02.423 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.405+0000 7fe8e70a9640 1 -- start start 2024-09-13T11:59:02.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.405+0000 7fe8e70a9640 10 monclient: init 2024-09-13T11:59:02.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e70a9640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:02.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e70a9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:59:02.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e70a9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8e009f6d0 0x7fe8e011db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e70a9640 10 monclient(hunting): picked mon.noname-b con 0x7fe8e009f6d0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:02.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e70a9640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8e00a15a0 0x7fe8e011e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e70a9640 10 monclient(hunting): picked mon.noname-c con 0x7fe8e00a15a0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:02.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e70a9640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e01343d0 0x7fe8e011e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e70a9640 10 monclient(hunting): picked mon.noname-a con 0x7fe8e01343d0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:02.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e70a9640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e70a9640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e009d120 con 0x7fe8e01343d0 2024-09-13T11:59:02.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e70a9640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.424 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e70a9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e009cfa0 con 0x7fe8e009f6d0 2024-09-13T11:59:02.425 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e70a9640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.425 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e70a9640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fe8e0134da0 con 0x7fe8e00a15a0 2024-09-13T11:59:02.425 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e70a9640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:02.425 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e70a9640 10 monclient(hunting): authenticate will time out at 1122.999375s 2024-09-13T11:59:02.425 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8dffff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8e00a15a0 0x7fe8e011e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.425 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8dffff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8e00a15a0 0x7fe8e011e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:51798/0 (socket says 172.21.15.100:51798) 2024-09-13T11:59:02.425 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e5646640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e01343d0 0x7fe8e011e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.425 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8dffff640 1 -- 172.21.15.100:0/326802105 learned_addr learned my addr 172.21.15.100:0/326802105 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:02.425 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8dffff640 10 monclient(hunting): get_auth_request con 0x7fe8e00a15a0 auth_method 0 2024-09-13T11:59:02.425 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.425 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8dffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.425 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e5646640 10 monclient(hunting): get_auth_request con 0x7fe8e01343d0 auth_method 0 2024-09-13T11:59:02.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e5646640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e5646640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e5646640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:02.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e5646640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e5646640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e5646640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:02.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e4e45640 1 --2- 172.21.15.100:0/326802105 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8e009f6d0 0x7fe8e011db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e4e45640 10 monclient(hunting): get_auth_request con 0x7fe8e009f6d0 auth_method 0 2024-09-13T11:59:02.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e4e45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.426 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e4e45640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e4e45640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e5646640 10 monclient(hunting): handle_auth_done global_id 15471 payload 824 2024-09-13T11:59:02.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e5646640 1 -- 172.21.15.100:0/326802105 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8e00a15a0 msgr2=0x7fe8e011e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e5646640 1 --2- 172.21.15.100:0/326802105 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8e00a15a0 0x7fe8e011e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e5646640 1 -- 172.21.15.100:0/326802105 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8e009f6d0 msgr2=0x7fe8e011db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e5646640 1 --2- 172.21.15.100:0/326802105 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8e009f6d0 0x7fe8e011db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e5646640 10 monclient: _finish_hunting 0 2024-09-13T11:59:02.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e5646640 1 monclient: found mon.noname-a 2024-09-13T11:59:02.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e5646640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:02.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e5646640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:02.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e5646640 1 -- 172.21.15.100:0/326802105 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8e0121580 con 0x7fe8e01343d0 2024-09-13T11:59:02.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8dffff640 1 --2- 172.21.15.100:0/326802105 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe8e00a15a0 0x7fe8e011e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:59:02.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.406+0000 7fe8e5646640 1 --2- 172.21.15.100:0/326802105 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e01343d0 0x7fe8e011e600 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe8d8030000 tx=0x7fe8d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:02.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8dd7fa640 1 -- 172.21.15.100:0/326802105 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d803f020 con 0x7fe8e01343d0 2024-09-13T11:59:02.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8dd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:02.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8dd7fa640 10 monclient: dump: 2024-09-13T11:59:02.428 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.428 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.428 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.428 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.428 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.428 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.428 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.428 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.428 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.429 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8dd7fa640 10 monclient: _finish_auth 0 2024-09-13T11:59:02.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8dd7fa640 10 monclient: _check_auth_tickets 2024-09-13T11:59:02.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:02.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8dd7fa640 1 -- 172.21.15.100:0/326802105 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8d8037040 con 0x7fe8e01343d0 2024-09-13T11:59:02.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8e70a9640 5 monclient: authenticate success, global_id 15471 2024-09-13T11:59:02.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8e70a9640 10 monclient: _renew_subs 2024-09-13T11:59:02.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8e70a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:02.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8e70a9640 1 -- 172.21.15.100:0/326802105 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8e0121860 con 0x7fe8e01343d0 2024-09-13T11:59:02.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8dd7fa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:02.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8dd7fa640 1 -- 172.21.15.100:0/326802105 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d8036070 con 0x7fe8e01343d0 2024-09-13T11:59:02.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8dd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:02.430 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8dd7fa640 10 monclient: dump: 2024-09-13T11:59:02.430 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.430 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.430 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.430 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.430 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.430 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.430 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.430 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.430 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.430 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.430 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8deffd640 4 set_mon_vals no callback set 2024-09-13T11:59:02.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8e70a9640 10 monclient: _renew_subs 2024-09-13T11:59:02.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8e70a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:02.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8e70a9640 1 -- 172.21.15.100:0/326802105 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8e00a4140 con 0x7fe8e01343d0 2024-09-13T11:59:02.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8e70a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:02.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8e70a9640 1 --2- 172.21.15.100:0/326802105 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e01c8c10 0x7fe8e002d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8e70a9640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.431 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.407+0000 7fe8e70a9640 1 -- 172.21.15.100:0/326802105 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e01209f0 con 0x7fe8e01c8c10 2024-09-13T11:59:02.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8e4e45640 1 --2- 172.21.15.100:0/326802105 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e01c8c10 0x7fe8e002d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8e4e45640 10 monclient: get_auth_request con 0x7fe8e01c8c10 auth_method 0 2024-09-13T11:59:02.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8e4e45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8e4e45640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8e4e45640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8dd7fa640 1 -- 172.21.15.100:0/326802105 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fe8d8042020 con 0x7fe8e01343d0 2024-09-13T11:59:02.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8e4e45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8e4e45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8e4e45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:02.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8e4e45640 10 monclient(hunting): handle_auth_done global_id 15474 payload 824 2024-09-13T11:59:02.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8e4e45640 1 -- 172.21.15.100:0/326802105 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe8e0134840 con 0x7fe8e01c8c10 2024-09-13T11:59:02.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8dd7fa640 1 --2- 172.21.15.100:0/326802105 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fe8c403dfd0 0x7fe8c4040490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.432 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8dd7fa640 1 -- 172.21.15.100:0/326802105 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7fe8d8072ae0 con 0x7fe8e01343d0 2024-09-13T11:59:02.433 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8e4e45640 1 --2- 172.21.15.100:0/326802105 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e01c8c10 0x7fe8e002d1f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fe8d400ba90 tx=0x7fe8d400bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:02.433 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8dd7fa640 1 -- 172.21.15.100:0/326802105 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d401d020 con 0x7fe8e01c8c10 2024-09-13T11:59:02.433 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.433 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8dd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:02.433 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8dd7fa640 10 monclient: dump: 2024-09-13T11:59:02.433 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.433 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.433 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.433 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.433 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.433 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.433 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.434 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.434 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.434 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8dffff640 1 --2- 172.21.15.100:0/326802105 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fe8c403dfd0 0x7fe8c4040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8dffff640 10 monclient: get_auth_request con 0x7fe8c403dfd0 auth_method 0 2024-09-13T11:59:02.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8dd7fa640 1 -- 172.21.15.100:0/326802105 <== mon.0 v2:172.21.15.100:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe8d4014e40 con 0x7fe8e01c8c10 2024-09-13T11:59:02.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8dd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:02.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8dd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:59:02.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8dd7fa640 1 -- 172.21.15.100:0/326802105 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e01c8c10 msgr2=0x7fe8e002d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8dd7fa640 1 --2- 172.21.15.100:0/326802105 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e01c8c10 0x7fe8e002d1f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fe8d400ba90 tx=0x7fe8d400bf50 comp rx=0 tx=0).stop 2024-09-13T11:59:02.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8dd7fa640 1 -- 172.21.15.100:0/326802105 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e01c8c10 msgr2=0x7fe8e002d1f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:59:02.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.408+0000 7fe8dd7fa640 1 --2- 172.21.15.100:0/326802105 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e01c8c10 0x7fe8e002d1f0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.434 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.409+0000 7fe8dffff640 1 --2- 172.21.15.100:0/326802105 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fe8c403dfd0 0x7fe8c4040490 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe8e011fff0 tx=0x7fe8cc008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:02.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.414+0000 7fe8e70a9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:59:02.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.414+0000 7fe8e70a9640 1 --2- 172.21.15.100:0/326802105 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e01c8ff0 0x7fe8e002d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.414+0000 7fe8e70a9640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.414+0000 7fe8e70a9640 1 -- 172.21.15.100:0/326802105 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e0072500 con 0x7fe8e01c8ff0 2024-09-13T11:59:02.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8e4e45640 1 --2- 172.21.15.100:0/326802105 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e01c8ff0 0x7fe8e002d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8e4e45640 10 monclient: get_auth_request con 0x7fe8e01c8ff0 auth_method 0 2024-09-13T11:59:02.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8e4e45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8e4e45640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8e4e45640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8e4e45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8e4e45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8e4e45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:02.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8e4e45640 10 monclient(hunting): handle_auth_done global_id 15477 payload 824 2024-09-13T11:59:02.437 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8e4e45640 1 -- 172.21.15.100:0/326802105 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe8e00008d0 con 0x7fe8e01c8ff0 2024-09-13T11:59:02.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8e4e45640 1 --2- 172.21.15.100:0/326802105 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e01c8ff0 0x7fe8e002d730 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe8d400ba90 tx=0x7fe8d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:02.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8dd7fa640 1 -- 172.21.15.100:0/326802105 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d401d030 con 0x7fe8e01c8ff0 2024-09-13T11:59:02.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8dd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:02.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8dd7fa640 10 monclient: dump: 2024-09-13T11:59:02.438 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.438 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.438 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.438 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.438 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.438 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.438 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.438 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.438 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.439 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8dd7fa640 1 -- 172.21.15.100:0/326802105 <== mon.0 v2:172.21.15.100:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4889 (secure 0 0 0) 0x7fe8d4024220 con 0x7fe8e01c8ff0 2024-09-13T11:59:02.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8dd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:59:02.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8dd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:59:02.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8dd7fa640 1 -- 172.21.15.100:0/326802105 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e01c8ff0 msgr2=0x7fe8e002d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8dd7fa640 1 --2- 172.21.15.100:0/326802105 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e01c8ff0 0x7fe8e002d730 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe8d400ba90 tx=0x7fe8d40046c0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8dd7fa640 1 -- 172.21.15.100:0/326802105 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e01c8ff0 msgr2=0x7fe8e002d730 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:59:02.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8dd7fa640 1 --2- 172.21.15.100:0/326802105 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e01c8ff0 0x7fe8e002d730 secure :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe8d400ba90 tx=0x7fe8d40046c0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.439 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:59:02.440 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:59:02.440 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:59:02.440 INFO:teuthology.orchestra.run.smithi100.stdout: "state": "leader", 2024-09-13T11:59:02.440 INFO:teuthology.orchestra.run.smithi100.stdout: "election_epoch": 28, 2024-09-13T11:59:02.440 INFO:teuthology.orchestra.run.smithi100.stdout: "uptime": 243414, 2024-09-13T11:59:02.440 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum": [ 2024-09-13T11:59:02.440 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-13T11:59:02.440 INFO:teuthology.orchestra.run.smithi100.stdout: 1, 2024-09-13T11:59:02.441 INFO:teuthology.orchestra.run.smithi100.stdout: 2 2024-09-13T11:59:02.441 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:59:02.441 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_age": 1, 2024-09-13T11:59:02.441 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:59:02.441 INFO:teuthology.orchestra.run.smithi100.stdout: "required_con": "2450521708007866372", 2024-09-13T11:59:02.441 INFO:teuthology.orchestra.run.smithi100.stdout: "required_mon": [ 2024-09-13T11:59:02.441 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:59:02.441 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:59:02.441 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:59:02.441 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:59:02.442 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:59:02.442 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:59:02.442 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:59:02.442 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:59:02.442 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:59:02.442 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:59:02.442 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:59:02.442 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:59:02.442 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:59:02.442 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_mon": [ 2024-09-13T11:59:02.442 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:59:02.442 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:59:02.443 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:59:02.443 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:59:02.443 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:59:02.443 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:59:02.443 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:59:02.443 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:59:02.443 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:59:02.443 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:59:02.443 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:59:02.443 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:59:02.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.443 INFO:teuthology.orchestra.run.smithi100.stdout: "outside_quorum": [], 2024-09-13T11:59:02.444 INFO:teuthology.orchestra.run.smithi100.stdout: "extra_probe_peers": [], 2024-09-13T11:59:02.444 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_provider": [], 2024-09-13T11:59:02.444 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": { 2024-09-13T11:59:02.444 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-09-13T11:59:02.444 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "0b1f4b4d-fad4-4dd3-86ac-a286d80846c0", 2024-09-13T11:59:02.444 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:59:02.444 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:59:02.444 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release": 19, 2024-09-13T11:59:02.444 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release_name": "squid", 2024-09-13T11:59:02.444 INFO:teuthology.orchestra.run.smithi100.stdout: "election_strategy": 1, 2024-09-13T11:59:02.444 INFO:teuthology.orchestra.run.smithi100.stdout: "disallowed_leaders: ": "", 2024-09-13T11:59:02.444 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false, 2024-09-13T11:59:02.444 INFO:teuthology.orchestra.run.smithi100.stdout: "tiebreaker_mon": "", 2024-09-13T11:59:02.445 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_ranks: ": "", 2024-09-13T11:59:02.445 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:59:02.445 INFO:teuthology.orchestra.run.smithi100.stdout: "persistent": [ 2024-09-13T11:59:02.445 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:59:02.445 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:59:02.445 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:59:02.445 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:59:02.445 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:59:02.445 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:59:02.445 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:59:02.445 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:59:02.445 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:59:02.446 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:59:02.446 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:59:02.446 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:59:02.446 INFO:teuthology.orchestra.run.smithi100.stdout: "optional": [] 2024-09-13T11:59:02.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.446 INFO:teuthology.orchestra.run.smithi100.stdout: "mons": [ 2024-09-13T11:59:02.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.446 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:59:02.446 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:59:02.446 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:59:02.446 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:59:02.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.446 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:59:02.446 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3300", 2024-09-13T11:59:02.447 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:59:02.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.447 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:59:02.447 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789", 2024-09-13T11:59:02.447 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:59:02.447 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:59:02.447 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:59:02.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.447 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789/0", 2024-09-13T11:59:02.447 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-13T11:59:02.447 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:59:02.448 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:59:02.448 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:59:02.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.448 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:59:02.448 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:59:02.448 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:59:02.448 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:59:02.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.448 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:59:02.448 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:3300", 2024-09-13T11:59:02.448 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:59:02.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.449 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:59:02.449 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789", 2024-09-13T11:59:02.449 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:59:02.449 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:59:02.449 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:59:02.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.449 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789/0", 2024-09-13T11:59:02.449 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.122:6789/0", 2024-09-13T11:59:02.449 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:59:02.449 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:59:02.449 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:59:02.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.450 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:59:02.450 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:59:02.450 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:59:02.450 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:59:02.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.450 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:59:02.450 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3301", 2024-09-13T11:59:02.450 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:59:02.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.450 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:59:02.450 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790", 2024-09-13T11:59:02.451 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:59:02.451 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:59:02.451 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:59:02.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.451 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790/0", 2024-09-13T11:59:02.451 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-13T11:59:02.451 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:59:02.451 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:59:02.451 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:59:02.451 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:59:02.451 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:59:02.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.451 INFO:teuthology.orchestra.run.smithi100.stdout: "feature_map": { 2024-09-13T11:59:02.452 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": [ 2024-09-13T11:59:02.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.452 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:59:02.452 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:59:02.452 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:59:02.452 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:59:02.452 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:59:02.452 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": [ 2024-09-13T11:59:02.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.453 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:59:02.453 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:59:02.453 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 3 2024-09-13T11:59:02.453 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:59:02.453 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:59:02.453 INFO:teuthology.orchestra.run.smithi100.stdout: "client": [ 2024-09-13T11:59:02.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.453 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:59:02.453 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:59:02.453 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 4 2024-09-13T11:59:02.453 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:59:02.454 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:59:02.454 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": [ 2024-09-13T11:59:02.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.454 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:59:02.454 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:59:02.454 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:59:02.454 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:59:02.454 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:59:02.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.454 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false 2024-09-13T11:59:02.454 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:59:02.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8e70a9640 1 -- 172.21.15.100:0/326802105 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fe8c403dfd0 msgr2=0x7fe8c4040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8e70a9640 1 --2- 172.21.15.100:0/326802105 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fe8c403dfd0 0x7fe8c4040490 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe8e011fff0 tx=0x7fe8cc008040 comp rx=0 tx=0).stop 2024-09-13T11:59:02.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.415+0000 7fe8e70a9640 10 monclient: shutdown 2024-09-13T11:59:02.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.416+0000 7fe8e70a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:02.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.416+0000 7fe8e70a9640 1 -- 172.21.15.100:0/326802105 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e01343d0 msgr2=0x7fe8e011e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.416+0000 7fe8e70a9640 1 --2- 172.21.15.100:0/326802105 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e01343d0 0x7fe8e011e600 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe8d8030000 tx=0x7fe8d8004a40 comp rx=0 tx=0).stop 2024-09-13T11:59:02.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.416+0000 7fe8e5646640 1 -- 172.21.15.100:0/326802105 reap_dead start 2024-09-13T11:59:02.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.416+0000 7fe8e70a9640 1 -- 172.21.15.100:0/326802105 shutdown_connections 2024-09-13T11:59:02.456 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.416+0000 7fe8e70a9640 1 --2- 172.21.15.100:0/326802105 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe8e01343d0 0x7fe8e011e600 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.416+0000 7fe8e70a9640 1 -- 172.21.15.100:0/326802105 >> 172.21.15.100:0/326802105 conn(0x7fe8e00a6470 msgr2=0x7fe8e0133050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:02.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.416+0000 7fe8e70a9640 1 -- 172.21.15.100:0/326802105 shutdown_connections 2024-09-13T11:59:02.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.416+0000 7fe8e70a9640 1 -- 172.21.15.100:0/326802105 wait complete. 2024-09-13T11:59:02.457 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T11:59:02.457 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-13T11:59:02.457 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T11:59:02.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.554+0000 7f2ce2840640 10 monclient: get_monmap_and_config 2024-09-13T11:59:02.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.554+0000 7f2ce2840640 10 monclient: build_initial_monmap 2024-09-13T11:59:02.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.554+0000 7f2ce2840640 10 monclient: monmap: 2024-09-13T11:59:02.555 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:02.555 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.555 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:02.555045+0000 2024-09-13T11:59:02.555 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:02.555045+0000 2024-09-13T11:59:02.555 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:02.556 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.556 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:02.556 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:02.556 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:02.556 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:02.556 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:02.556 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.554+0000 7f2ce2840640 1 Processor -- start 2024-09-13T11:59:02.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.554+0000 7f2ce2840640 1 -- start start 2024-09-13T11:59:02.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.554+0000 7f2ce2840640 10 monclient: init 2024-09-13T11:59:02.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce2840640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:02.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce2840640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-13T11:59:02.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce2840640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2cdc0a15a0 addr v1:172.21.15.122:6789/0 2024-09-13T11:59:02.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce2840640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc09f6d0 0x7f2cdc139680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce2840640 10 monclient(hunting): picked mon.noname-b con 0x7f2cdc09f6d0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:02.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce2840640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f2cdc0a2090 0x7f2cdc13bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce2840640 10 monclient(hunting): picked mon.noname-a con 0x7f2cdc0a2090 addr v2:172.21.15.100:3300/0 2024-09-13T11:59:02.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce2840640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cdc09ecf0 con 0x7f2cdc0a15a0 2024-09-13T11:59:02.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce2840640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce2840640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f2cdc09d3b0 con 0x7f2cdc0a2090 2024-09-13T11:59:02.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce2840640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce2840640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f2cdc09ea10 con 0x7f2cdc09f6d0 2024-09-13T11:59:02.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce2840640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:02.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce2840640 10 monclient(hunting): authenticate will time out at 1123.148493s 2024-09-13T11:59:02.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2cdb7fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc09f6d0 0x7f2cdc139680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce0ddd640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f2cdc0a2090 0x7f2cdc13bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2cdb7fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc09f6d0 0x7f2cdc139680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:51814/0 (socket says 172.21.15.100:51814) 2024-09-13T11:59:02.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2cdb7fe640 1 -- 172.21.15.100:0/2328520205 learned_addr learned my addr 172.21.15.100:0/2328520205 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:02.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce0ddd640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f2cdc0a2090 0x7f2cdc13bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47062/0 (socket says 172.21.15.100:47062) 2024-09-13T11:59:02.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce0ddd640 10 monclient(hunting): get_auth_request con 0x7f2cdc0a2090 auth_method 0 2024-09-13T11:59:02.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce0ddd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce0ddd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce0ddd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2cdb7fe640 10 monclient(hunting): get_auth_request con 0x7f2cdc09f6d0 auth_method 0 2024-09-13T11:59:02.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2cdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2cdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2cdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce0ddd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce0ddd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce0ddd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:02.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2cdb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2cdb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2cdb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:02.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce0ddd640 10 monclient(hunting): handle_auth_done global_id 15480 payload 274 2024-09-13T11:59:02.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce0ddd640 1 -- 172.21.15.100:0/2328520205 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc09f6d0 msgr2=0x7f2cdc139680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:02.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce0ddd640 1 --2- 172.21.15.100:0/2328520205 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc09f6d0 0x7f2cdc139680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce0ddd640 1 -- 172.21.15.100:0/2328520205 >> v1:172.21.15.122:6789/0 conn(0x7f2cdc0a15a0 legacy=0x7f2cdc0a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:02.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce0ddd640 10 monclient: _finish_hunting 0 2024-09-13T11:59:02.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce0ddd640 1 monclient: found mon.noname-a 2024-09-13T11:59:02.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce0ddd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:02.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce0ddd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:02.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2ce0ddd640 1 -- 172.21.15.100:0/2328520205 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cdc0a3250 con 0x7f2cdc0a2090 2024-09-13T11:59:02.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.555+0000 7f2cdb7fe640 1 --2- 172.21.15.100:0/2328520205 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc09f6d0 0x7f2cdc139680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:59:02.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.556+0000 7f2ce0ddd640 1 --2- 172.21.15.100:0/2328520205 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2cdc0a2090 0x7f2cdc13bbd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4009d90 tx=0x7f2cd402a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1821489701c49b0d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:02.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.556+0000 7f2cdaffd640 1 -- 172.21.15.100:0/2328520205 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4007740 con 0x7f2cdc0a2090 2024-09-13T11:59:02.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.556+0000 7f2cdaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.556+0000 7f2cdaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:02.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.556+0000 7f2cdaffd640 10 monclient: dump: 2024-09-13T11:59:02.570 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.570 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.570 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.570 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.570 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.570 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.571 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.571 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.571 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.571 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.556+0000 7f2cdaffd640 10 monclient: _finish_auth 0 2024-09-13T11:59:02.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.556+0000 7f2cdaffd640 10 monclient: _check_auth_tickets 2024-09-13T11:59:02.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.556+0000 7f2cdaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:02.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.556+0000 7f2cdaffd640 1 -- 172.21.15.100:0/2328520205 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cd40078e0 con 0x7f2cdc0a2090 2024-09-13T11:59:02.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.556+0000 7f2ce2840640 5 monclient: authenticate success, global_id 15480 2024-09-13T11:59:02.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.556+0000 7f2ce2840640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:02.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.556+0000 7f2cdaffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:02.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.556+0000 7f2ce2840640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:02.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.556+0000 7f2cdaffd640 1 -- 172.21.15.100:0/2328520205 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4007cb0 con 0x7f2cdc0a2090 2024-09-13T11:59:02.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.556+0000 7f2cdaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.556+0000 7f2ce2840640 4 set_mon_vals no callback set 2024-09-13T11:59:02.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.556+0000 7f2cdaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:02.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.556+0000 7f2cdaffd640 10 monclient: dump: 2024-09-13T11:59:02.572 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.572 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.572 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.572 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.572 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.572 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.573 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.573 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.573 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.573 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.557+0000 7f2ce2840640 10 monclient: shutdown 2024-09-13T11:59:02.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.557+0000 7f2ce2840640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:02.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.557+0000 7f2ce2840640 1 -- 172.21.15.100:0/2328520205 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2cdc0a2090 msgr2=0x7f2cdc13bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.557+0000 7f2ce2840640 1 --2- 172.21.15.100:0/2328520205 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2cdc0a2090 0x7f2cdc13bbd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4009d90 tx=0x7f2cd402a490 comp rx=0 tx=0).stop 2024-09-13T11:59:02.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.557+0000 7f2ce2840640 1 -- 172.21.15.100:0/2328520205 shutdown_connections 2024-09-13T11:59:02.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.557+0000 7f2ce2840640 1 --2- 172.21.15.100:0/2328520205 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2cdc0a2090 0x7f2cdc13bbd0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.557+0000 7f2ce2840640 1 --2- 172.21.15.100:0/2328520205 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc09f6d0 0x7f2cdc139680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.557+0000 7f2ce2840640 1 -- 172.21.15.100:0/2328520205 >> 172.21.15.100:0/2328520205 conn(0x7f2cdc0a6470 msgr2=0x7f2cdc0a2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:02.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.557+0000 7f2ce2840640 1 -- 172.21.15.100:0/2328520205 shutdown_connections 2024-09-13T11:59:02.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.557+0000 7f2ce2840640 1 -- 172.21.15.100:0/2328520205 wait complete. 2024-09-13T11:59:02.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.558+0000 7f2ce2840640 10 monclient: build_initial_monmap 2024-09-13T11:59:02.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.558+0000 7f2ce2840640 10 monclient: monmap: 2024-09-13T11:59:02.574 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:02.574 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:02.574 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:02.574 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:02.574 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:02.574 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.574 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:02.574 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:02.574 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:02.574 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.558+0000 7f2ce2840640 1 Processor -- start 2024-09-13T11:59:02.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.558+0000 7f2ce2840640 1 -- start start 2024-09-13T11:59:02.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.558+0000 7f2ce2840640 10 monclient: init 2024-09-13T11:59:02.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.558+0000 7f2ce2840640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:02.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.558+0000 7f2ce2840640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:59:02.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.558+0000 7f2ce2840640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cdc09f6d0 0x7f2cdc1c88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.558+0000 7f2ce2840640 10 monclient(hunting): picked mon.noname-b con 0x7f2cdc09f6d0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:02.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.558+0000 7f2ce2840640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2cdc0a15a0 0x7f2cdc1c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce2840640 10 monclient(hunting): picked mon.noname-a con 0x7f2cdc0a15a0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:02.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce2840640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc0a2090 0x7f2cdc120c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce2840640 10 monclient(hunting): picked mon.noname-c con 0x7f2cdc0a2090 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:02.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce2840640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce2840640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f2cdc09ba00 con 0x7f2cdc0a15a0 2024-09-13T11:59:02.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce2840640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce2840640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2cdc09d1b0 con 0x7f2cdc09f6d0 2024-09-13T11:59:02.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce2840640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce2840640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f2cdc09bb80 con 0x7f2cdc0a2090 2024-09-13T11:59:02.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce2840640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:02.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce2840640 10 monclient(hunting): authenticate will time out at 1123.152031s 2024-09-13T11:59:02.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdb7fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2cdc0a15a0 0x7f2cdc1c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce0ddd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc0a2090 0x7f2cdc120c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce0ddd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc0a2090 0x7f2cdc120c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:51830/0 (socket says 172.21.15.100:51830) 2024-09-13T11:59:02.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce0ddd640 1 -- 172.21.15.100:0/960854121 learned_addr learned my addr 172.21.15.100:0/960854121 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:02.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce0ddd640 10 monclient(hunting): get_auth_request con 0x7f2cdc0a2090 auth_method 0 2024-09-13T11:59:02.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdb7fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2cdc0a15a0 0x7f2cdc1c8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47066/0 (socket says 172.21.15.100:47066) 2024-09-13T11:59:02.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce0ddd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce0ddd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce0ddd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdb7fe640 10 monclient(hunting): get_auth_request con 0x7f2cdc0a15a0 auth_method 0 2024-09-13T11:59:02.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:02.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce0ddd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce0ddd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce0ddd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:02.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdbfff640 1 --2- 172.21.15.100:0/960854121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cdc09f6d0 0x7f2cdc1c88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdbfff640 10 monclient(hunting): get_auth_request con 0x7f2cdc09f6d0 auth_method 0 2024-09-13T11:59:02.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdbfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdb7fe640 10 monclient(hunting): handle_auth_done global_id 15483 payload 824 2024-09-13T11:59:02.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdb7fe640 1 -- 172.21.15.100:0/960854121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc0a2090 msgr2=0x7f2cdc120c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdb7fe640 1 --2- 172.21.15.100:0/960854121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc0a2090 0x7f2cdc120c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.578 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdb7fe640 1 -- 172.21.15.100:0/960854121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cdc09f6d0 msgr2=0x7f2cdc1c88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdb7fe640 1 --2- 172.21.15.100:0/960854121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cdc09f6d0 0x7f2cdc1c88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdb7fe640 10 monclient: _finish_hunting 0 2024-09-13T11:59:02.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdb7fe640 1 monclient: found mon.noname-a 2024-09-13T11:59:02.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:02.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:02.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdb7fe640 1 -- 172.21.15.100:0/960854121 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cdc135030 con 0x7f2cdc0a15a0 2024-09-13T11:59:02.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce0ddd640 1 --2- 172.21.15.100:0/960854121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc0a2090 0x7f2cdc120c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:59:02.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cdb7fe640 1 --2- 172.21.15.100:0/960854121 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2cdc0a15a0 0x7f2cdc1c8de0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f2cc400e930 tx=0x7f2cc400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:02.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cd8ff9640 1 -- 172.21.15.100:0/960854121 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc4004030 con 0x7f2cdc0a15a0 2024-09-13T11:59:02.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cd8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cd8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:02.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cd8ff9640 10 monclient: dump: 2024-09-13T11:59:02.579 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.580 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.580 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.580 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.580 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.580 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.580 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.580 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.580 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.580 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cd8ff9640 10 monclient: _finish_auth 0 2024-09-13T11:59:02.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cd8ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:59:02.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cd8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:02.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce2840640 5 monclient: authenticate success, global_id 15483 2024-09-13T11:59:02.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cd8ff9640 1 -- 172.21.15.100:0/960854121 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cc40041d0 con 0x7f2cdc0a15a0 2024-09-13T11:59:02.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cd8ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:02.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cd8ff9640 1 -- 172.21.15.100:0/960854121 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc400c9a0 con 0x7f2cdc0a15a0 2024-09-13T11:59:02.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cd8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cd8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:02.581 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cd8ff9640 10 monclient: dump: 2024-09-13T11:59:02.581 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.581 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.581 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.581 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.581 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.581 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.581 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.581 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.582 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.582 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce2840640 10 monclient: _renew_subs 2024-09-13T11:59:02.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2cda7fc640 4 set_mon_vals no callback set 2024-09-13T11:59:02.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce2840640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:02.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce2840640 1 -- 172.21.15.100:0/960854121 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cdc124a60 con 0x7f2cdc0a15a0 2024-09-13T11:59:02.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce2840640 10 monclient: _renew_subs 2024-09-13T11:59:02.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce2840640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:02.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.559+0000 7f2ce2840640 1 -- 172.21.15.100:0/960854121 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cdc122700 con 0x7f2cdc0a15a0 2024-09-13T11:59:02.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.560+0000 7f2ce2840640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:02.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.560+0000 7f2ce2840640 1 --2- 172.21.15.100:0/960854121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc063ab0 0x7f2cdc063e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.560+0000 7f2ce2840640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.560+0000 7f2ce2840640 1 -- 172.21.15.100:0/960854121 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f2cdc0a1980 con 0x7f2cdc063ab0 2024-09-13T11:59:02.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.560+0000 7f2cdbfff640 1 --2- 172.21.15.100:0/960854121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc063ab0 0x7f2cdc063e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.560+0000 7f2cdbfff640 10 monclient: get_auth_request con 0x7f2cdc063ab0 auth_method 0 2024-09-13T11:59:02.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.560+0000 7f2cdbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.560+0000 7f2cdbfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.560+0000 7f2cdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.560+0000 7f2cd8ff9640 1 -- 172.21.15.100:0/960854121 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f2cc400cb40 con 0x7f2cdc0a15a0 2024-09-13T11:59:02.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.560+0000 7f2cdbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.560+0000 7f2cdbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.560+0000 7f2cdbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:02.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.560+0000 7f2cdbfff640 10 monclient(hunting): handle_auth_done global_id 24578 payload 824 2024-09-13T11:59:02.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.560+0000 7f2cd8ff9640 1 --2- 172.21.15.100:0/960854121 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f2cbc03dfa0 0x7f2cbc040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.560+0000 7f2cdbfff640 1 -- 172.21.15.100:0/960854121 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2cdc123260 con 0x7f2cdc063ab0 2024-09-13T11:59:02.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.560+0000 7f2cd8ff9640 1 -- 172.21.15.100:0/960854121 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7f2cc4052580 con 0x7f2cdc0a15a0 2024-09-13T11:59:02.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.561+0000 7f2cdbfff640 1 --2- 172.21.15.100:0/960854121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc063ab0 0x7f2cdc063e90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc00e4d0 tx=0x7f2ccc00e990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:02.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.561+0000 7f2ce0ddd640 1 --2- 172.21.15.100:0/960854121 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f2cbc03dfa0 0x7f2cbc040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.561+0000 7f2ce0ddd640 10 monclient: get_auth_request con 0x7f2cbc03dfa0 auth_method 0 2024-09-13T11:59:02.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.561+0000 7f2cd8ff9640 1 -- 172.21.15.100:0/960854121 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc007240 con 0x7f2cdc063ab0 2024-09-13T11:59:02.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.561+0000 7f2cd8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.561+0000 7f2cd8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:59:02.584 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.561+0000 7f2cd8ff9640 10 monclient: dump: 2024-09-13T11:59:02.584 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.584 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.584 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.584 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.584 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.585 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.585 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.585 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.585 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.585 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.561+0000 7f2cd8ff9640 1 -- 172.21.15.100:0/960854121 <== mon.2 v2:172.21.15.100:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2ccc007240 con 0x7f2cdc063ab0 2024-09-13T11:59:02.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.561+0000 7f2cd8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:02.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.561+0000 7f2cd8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:59:02.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.561+0000 7f2cd8ff9640 1 -- 172.21.15.100:0/960854121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc063ab0 msgr2=0x7f2cdc063e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.561+0000 7f2cd8ff9640 1 --2- 172.21.15.100:0/960854121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc063ab0 0x7f2cdc063e90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc00e4d0 tx=0x7f2ccc00e990 comp rx=0 tx=0).stop 2024-09-13T11:59:02.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.561+0000 7f2cd8ff9640 1 -- 172.21.15.100:0/960854121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc063ab0 msgr2=0x7f2cdc063e90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:59:02.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.561+0000 7f2cd8ff9640 1 --2- 172.21.15.100:0/960854121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc063ab0 0x7f2cdc063e90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.561+0000 7f2ce0ddd640 1 --2- 172.21.15.100:0/960854121 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f2cbc03dfa0 0x7f2cbc040460 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4009bd0 tx=0x7f2cd402f000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:02.587 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.567+0000 7f2ce2840640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:59:02.587 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.567+0000 7f2ce2840640 1 --2- 172.21.15.100:0/960854121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc02d1f0 0x7f2cdc02d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.587 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.567+0000 7f2ce2840640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.587 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.567+0000 7f2ce2840640 1 -- 172.21.15.100:0/960854121 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f2cdc11d8f0 con 0x7f2cdc02d1f0 2024-09-13T11:59:02.588 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.567+0000 7f2cdbfff640 1 --2- 172.21.15.100:0/960854121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc02d1f0 0x7f2cdc02d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.588 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.567+0000 7f2cdbfff640 10 monclient: get_auth_request con 0x7f2cdc02d1f0 auth_method 0 2024-09-13T11:59:02.588 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.567+0000 7f2cdbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.588 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.567+0000 7f2cdbfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.588 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.567+0000 7f2cdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.588 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.567+0000 7f2cdbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.588 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.567+0000 7f2cdbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.588 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.567+0000 7f2cdbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:02.588 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.567+0000 7f2cdbfff640 10 monclient(hunting): handle_auth_done global_id 24581 payload 824 2024-09-13T11:59:02.588 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.567+0000 7f2cdbfff640 1 -- 172.21.15.100:0/960854121 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2cdc06ca60 con 0x7f2cdc02d1f0 2024-09-13T11:59:02.588 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.567+0000 7f2cdbfff640 1 --2- 172.21.15.100:0/960854121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc02d1f0 0x7f2cdc02d600 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc00e4d0 tx=0x7f2ccc003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:02.588 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.568+0000 7f2cd8ff9640 1 -- 172.21.15.100:0/960854121 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc01eb70 con 0x7f2cdc02d1f0 2024-09-13T11:59:02.588 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.568+0000 7f2cd8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.589 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.568+0000 7f2cd8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:59:02.589 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.568+0000 7f2cd8ff9640 10 monclient: dump: 2024-09-13T11:59:02.589 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.589 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.589 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.589 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.589 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.589 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.589 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.589 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.589 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.589 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.589 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.568+0000 7f2cd8ff9640 1 -- 172.21.15.100:0/960854121 <== mon.2 v2:172.21.15.100:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f2ccc01eb70 con 0x7f2cdc02d1f0 2024-09-13T11:59:02.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.568+0000 7f2cd8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:59:02.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.568+0000 7f2cd8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:59:02.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.568+0000 7f2cd8ff9640 1 -- 172.21.15.100:0/960854121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc02d1f0 msgr2=0x7f2cdc02d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.568+0000 7f2cd8ff9640 1 --2- 172.21.15.100:0/960854121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc02d1f0 0x7f2cdc02d600 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc00e4d0 tx=0x7f2ccc003760 comp rx=0 tx=0).stop 2024-09-13T11:59:02.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.568+0000 7f2cd8ff9640 1 -- 172.21.15.100:0/960854121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc02d1f0 msgr2=0x7f2cdc02d600 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:59:02.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.568+0000 7f2cd8ff9640 1 --2- 172.21.15.100:0/960854121 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2cdc02d1f0 0x7f2cdc02d600 secure :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc00e4d0 tx=0x7f2ccc003760 comp rx=0 tx=0).stop 2024-09-13T11:59:02.590 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:59:02.590 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:59:02.590 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:59:02.590 INFO:teuthology.orchestra.run.smithi100.stdout: "state": "peon", 2024-09-13T11:59:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "election_epoch": 28, 2024-09-13T11:59:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "uptime": 135545, 2024-09-13T11:59:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum": [ 2024-09-13T11:59:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-13T11:59:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: 1, 2024-09-13T11:59:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: 2 2024-09-13T11:59:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:59:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_age": 2, 2024-09-13T11:59:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:59:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "required_con": "2450521708007866372", 2024-09-13T11:59:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "required_mon": [ 2024-09-13T11:59:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:59:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:59:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:59:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:59:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:59:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:59:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:59:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:59:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:59:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:59:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:59:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:59:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:59:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_mon": [ 2024-09-13T11:59:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:59:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:59:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:59:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:59:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:59:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:59:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:59:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:59:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:59:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:59:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:59:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:59:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "outside_quorum": [], 2024-09-13T11:59:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "extra_probe_peers": [], 2024-09-13T11:59:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_provider": [], 2024-09-13T11:59:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": { 2024-09-13T11:59:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-09-13T11:59:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "0b1f4b4d-fad4-4dd3-86ac-a286d80846c0", 2024-09-13T11:59:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:59:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:59:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release": 19, 2024-09-13T11:59:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release_name": "squid", 2024-09-13T11:59:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "election_strategy": 1, 2024-09-13T11:59:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "disallowed_leaders: ": "", 2024-09-13T11:59:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false, 2024-09-13T11:59:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "tiebreaker_mon": "", 2024-09-13T11:59:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_ranks: ": "", 2024-09-13T11:59:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:59:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "persistent": [ 2024-09-13T11:59:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:59:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:59:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:59:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:59:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:59:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:59:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:59:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:59:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:59:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:59:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:59:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:59:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "optional": [] 2024-09-13T11:59:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "mons": [ 2024-09-13T11:59:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:59:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:59:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:59:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:59:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:59:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3300", 2024-09-13T11:59:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:59:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:59:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789", 2024-09-13T11:59:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:59:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:59:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:59:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789/0", 2024-09-13T11:59:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-13T11:59:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:59:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:59:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:59:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:59:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:59:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:59:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:59:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:59:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:3300", 2024-09-13T11:59:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:59:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:59:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789", 2024-09-13T11:59:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:59:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:59:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:59:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789/0", 2024-09-13T11:59:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.122:6789/0", 2024-09-13T11:59:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:59:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:59:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:59:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:59:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:59:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:59:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:59:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:59:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3301", 2024-09-13T11:59:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:59:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:59:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790", 2024-09-13T11:59:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:59:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:59:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:59:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790/0", 2024-09-13T11:59:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-13T11:59:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:59:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:59:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:59:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:59:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:59:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "feature_map": { 2024-09-13T11:59:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": [ 2024-09-13T11:59:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:59:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:59:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:59:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:59:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:59:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": [ 2024-09-13T11:59:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:59:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:59:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 3 2024-09-13T11:59:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:59:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:59:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "client": [ 2024-09-13T11:59:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:59:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:59:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 3 2024-09-13T11:59:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:59:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:59:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false 2024-09-13T11:59:02.607 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:59:02.608 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.568+0000 7f2ce2840640 1 -- 172.21.15.100:0/960854121 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f2cbc03dfa0 msgr2=0x7f2cbc040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.608 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.568+0000 7f2ce2840640 1 --2- 172.21.15.100:0/960854121 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f2cbc03dfa0 0x7f2cbc040460 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4009bd0 tx=0x7f2cd402f000 comp rx=0 tx=0).stop 2024-09-13T11:59:02.608 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.568+0000 7f2ce2840640 10 monclient: shutdown 2024-09-13T11:59:02.608 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.568+0000 7f2ce2840640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:02.608 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.568+0000 7f2ce2840640 1 -- 172.21.15.100:0/960854121 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2cdc0a15a0 msgr2=0x7f2cdc1c8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.608 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.568+0000 7f2ce2840640 1 --2- 172.21.15.100:0/960854121 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2cdc0a15a0 0x7f2cdc1c8de0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f2cc400e930 tx=0x7f2cc400edf0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.608 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.568+0000 7f2ce0ddd640 1 -- 172.21.15.100:0/960854121 reap_dead start 2024-09-13T11:59:02.608 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.568+0000 7f2ce2840640 1 -- 172.21.15.100:0/960854121 shutdown_connections 2024-09-13T11:59:02.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.568+0000 7f2ce2840640 1 --2- 172.21.15.100:0/960854121 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2cdc0a15a0 0x7f2cdc1c8de0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.568+0000 7f2ce2840640 1 -- 172.21.15.100:0/960854121 >> 172.21.15.100:0/960854121 conn(0x7f2cdc0a6470 msgr2=0x7f2cdc071b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:02.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.568+0000 7f2ce2840640 1 -- 172.21.15.100:0/960854121 shutdown_connections 2024-09-13T11:59:02.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.568+0000 7f2ce2840640 1 -- 172.21.15.100:0/960854121 wait complete. 2024-09-13T11:59:02.609 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T11:59:02.609 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-13T11:59:02.609 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-13T11:59:02.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.706+0000 7f272a8b8640 10 monclient: get_monmap_and_config 2024-09-13T11:59:02.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.706+0000 7f272a8b8640 10 monclient: build_initial_monmap 2024-09-13T11:59:02.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.706+0000 7f272a8b8640 10 monclient: monmap: 2024-09-13T11:59:02.708 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:02.708 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.708 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:02.707567+0000 2024-09-13T11:59:02.708 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:02.707567+0000 2024-09-13T11:59:02.708 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:02.708 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.708 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:02.708 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:02.708 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:02.708 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:02.708 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:02.709 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.707+0000 7f272a8b8640 1 Processor -- start 2024-09-13T11:59:02.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.707+0000 7f272a8b8640 1 -- start start 2024-09-13T11:59:02.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.707+0000 7f272a8b8640 10 monclient: init 2024-09-13T11:59:02.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.707+0000 7f272a8b8640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:02.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.707+0000 7f272a8b8640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-13T11:59:02.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f272a8b8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f27240a37d0 addr v1:172.21.15.100:6789/0 2024-09-13T11:59:02.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f272a8b8640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f272409fd90 0x7f272413c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f272a8b8640 10 monclient(hunting): picked mon.noname-c con 0x7f272409fd90 addr v2:172.21.15.122:3300/0 2024-09-13T11:59:02.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f272a8b8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f27240a42c0 addr v1:172.21.15.122:6789/0 2024-09-13T11:59:02.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f272a8b8640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f272406fe10 con 0x7f27240a37d0 2024-09-13T11:59:02.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f272a8b8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f272406eff0 con 0x7f27240a42c0 2024-09-13T11:59:02.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f272a8b8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f272a8b8640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f272406f270 con 0x7f272409fd90 2024-09-13T11:59:02.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f272a8b8640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:02.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f272a8b8640 10 monclient(hunting): authenticate will time out at 1123.301074s 2024-09-13T11:59:02.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2723fff640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f27240a37d0 0x7f27240a3bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:57060/0 (socket says 172.21.15.100:57060) 2024-09-13T11:59:02.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2723fff640 1 -- 172.21.15.100:0/47645229 learned_addr learned my addr 172.21.15.100:0/47645229 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:59:02.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 1 -- 172.21.15.100:0/47645229 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 723247170 0 0) 0x7f272406fe10 con 0x7f27240a37d0 2024-09-13T11:59:02.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 10 monclient(hunting): _negotiate 2024-09-13T11:59:02.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 10 monclient(hunting): my global_id is 15486 2024-09-13T11:59:02.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 1 -- 172.21.15.100:0/47645229 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f270c003520 con 0x7f27240a37d0 2024-09-13T11:59:02.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f27237fe640 1 -- 172.21.15.100:0/47645229 >> v2:172.21.15.122:3300/0 conn(0x7f272409fd90 msgr2=0x7f272413c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T11:59:02.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f27237fe640 1 -- 172.21.15.100:0/47645229 >> v2:172.21.15.122:3300/0 conn(0x7f272409fd90 msgr2=0x7f272413c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:59:02.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f27237fe640 1 --2- 172.21.15.100:0/47645229 >> v2:172.21.15.122:3300/0 conn(0x7f272409fd90 0x7f272413c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2024-09-13T11:59:02.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f27237fe640 1 --2- 172.21.15.100:0/47645229 >> v2:172.21.15.122:3300/0 conn(0x7f272409fd90 0x7f272413c9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:59:02.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 1 -- 172.21.15.100:0/47645229 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1592813535 0 0) 0x7f270c003520 con 0x7f27240a37d0 2024-09-13T11:59:02.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 1 -- 172.21.15.100:0/47645229 >> v2:172.21.15.122:3300/0 conn(0x7f272409fd90 msgr2=0x7f272413c9a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:59:02.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 1 --2- 172.21.15.100:0/47645229 >> v2:172.21.15.122:3300/0 conn(0x7f272409fd90 0x7f272413c9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 1 -- 172.21.15.100:0/47645229 >> v1:172.21.15.122:6789/0 conn(0x7f27240a42c0 legacy=0x7f272413eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 10 monclient: _finish_hunting 0 2024-09-13T11:59:02.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 1 monclient: found mon.noname-a-legacy 2024-09-13T11:59:02.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:02.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.100:6789/0 2024-09-13T11:59:02.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 1 -- 172.21.15.100:0/47645229 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2724070310 con 0x7f27240a37d0 2024-09-13T11:59:02.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 10 monclient: _finish_auth 0 2024-09-13T11:59:02.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:59:02.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:02.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 1 -- 172.21.15.100:0/47645229 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 626700363 0 0) 0x7f27140016f0 con 0x7f27240a37d0 2024-09-13T11:59:02.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f272a8b8640 5 monclient: authenticate success, global_id 15486 2024-09-13T11:59:02.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f272a8b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:02.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-13T11:59:02.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 10 monclient: dump: 2024-09-13T11:59:02.723 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.723 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.723 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.724 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.724 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.724 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.724 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.724 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.724 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.724 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] but i'm connected to v1:172.21.15.100:6789/0, reconnecting 2024-09-13T11:59:02.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:02.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 1 -- 172.21.15.100:0/47645229 >> v1:172.21.15.100:6789/0 conn(0x7f27240a37d0 legacy=0x7f27240a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:59:02.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 1 --2- 172.21.15.100:0/47645229 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f270c003a50 0x7f270c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 10 monclient(hunting): picked mon.a con 0x7f270c003a50 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:02.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 1 --2- 172.21.15.100:0/47645229 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f270c004800 0x7f270c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 10 monclient(hunting): picked mon.c con 0x7f270c004800 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:02.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 1 --2- 172.21.15.100:0/47645229 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f270c0099b0 0x7f270c00bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 10 monclient(hunting): picked mon.b con 0x7f270c0099b0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:02.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 1 -- 172.21.15.100:0/47645229 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f272406f270 con 0x7f270c003a50 2024-09-13T11:59:02.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 1 -- 172.21.15.100:0/47645229 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f270c00c3c0 con 0x7f270c0099b0 2024-09-13T11:59:02.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 1 -- 172.21.15.100:0/47645229 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f270c00c5c0 con 0x7f270c004800 2024-09-13T11:59:02.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:02.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2722ffd640 10 monclient(hunting): _finish_auth 0 2024-09-13T11:59:02.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f272a8b8640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:02.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2723fff640 1 --2- 172.21.15.100:0/47645229 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f270c003a50 0x7f270c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2723fff640 10 monclient(hunting): get_auth_request con 0x7f270c003a50 auth_method 0 2024-09-13T11:59:02.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2723fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2723fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2723fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:59:02.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f27237fe640 1 --2- 172.21.15.100:0/47645229 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f270c004800 0x7f270c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f27237fe640 10 monclient(hunting): get_auth_request con 0x7f270c004800 auth_method 0 2024-09-13T11:59:02.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f27237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f27237fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f27237fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:59:02.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2723fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2723fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2723fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:59:02.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.708+0000 7f2728e55640 1 --2- 172.21.15.100:0/47645229 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f270c0099b0 0x7f270c00bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f27237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f27237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f27237fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:59:02.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2728e55640 10 monclient(hunting): get_auth_request con 0x7f270c0099b0 auth_method 0 2024-09-13T11:59:02.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2728e55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2728e55640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2728e55640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:59:02.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2723fff640 10 monclient(hunting): handle_auth_done global_id 15486 payload 293 2024-09-13T11:59:02.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2723fff640 1 -- 172.21.15.100:0/47645229 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f270c004800 msgr2=0x7f270c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:02.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2723fff640 1 --2- 172.21.15.100:0/47645229 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f270c004800 0x7f270c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2723fff640 1 -- 172.21.15.100:0/47645229 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f270c0099b0 msgr2=0x7f270c00bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:02.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2723fff640 1 --2- 172.21.15.100:0/47645229 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f270c0099b0 0x7f270c00bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2723fff640 10 monclient: _finish_hunting 0 2024-09-13T11:59:02.728 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2723fff640 1 monclient: found mon.a 2024-09-13T11:59:02.728 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2723fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:02.728 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2723fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:02.728 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2728e55640 1 -- 172.21.15.100:0/47645229 reap_dead start 2024-09-13T11:59:02.728 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2723fff640 1 -- 172.21.15.100:0/47645229 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f270c003520 con 0x7f270c003a50 2024-09-13T11:59:02.728 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2723fff640 10 monclient: _finish_auth 0 2024-09-13T11:59:02.728 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2723fff640 10 monclient: _check_auth_tickets 2024-09-13T11:59:02.728 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2723fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:02.728 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2723fff640 1 --2- 172.21.15.100:0/47645229 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f270c003a50 0x7f270c003e70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f271400c210 tx=0x7f270c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1eabc1a0c1f77cc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:02.728 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2722ffd640 1 -- 172.21.15.100:0/47645229 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2714007d30 con 0x7f270c003a50 2024-09-13T11:59:02.728 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2722ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.728 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2722ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:02.728 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2722ffd640 10 monclient: dump: 2024-09-13T11:59:02.729 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.729 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.729 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.729 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.729 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.729 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.729 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.729 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.729 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.729 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.729 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2722ffd640 1 -- 172.21.15.100:0/47645229 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2714005af0 con 0x7f270c003a50 2024-09-13T11:59:02.729 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2722ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:02.729 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2722ffd640 1 -- 172.21.15.100:0/47645229 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2714007070 con 0x7f270c003a50 2024-09-13T11:59:02.729 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2722ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.730 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2722ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:02.730 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f2722ffd640 10 monclient: dump: 2024-09-13T11:59:02.730 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.730 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.730 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.730 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.730 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.730 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.730 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.730 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.730 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.730 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.730 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f272a8b8640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:02.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.709+0000 7f272a8b8640 4 set_mon_vals no callback set 2024-09-13T11:59:02.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.710+0000 7f272a8b8640 10 monclient: shutdown 2024-09-13T11:59:02.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.710+0000 7f272a8b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:02.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.710+0000 7f272a8b8640 1 -- 172.21.15.100:0/47645229 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f270c003a50 msgr2=0x7f270c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.710+0000 7f272a8b8640 1 --2- 172.21.15.100:0/47645229 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f270c003a50 0x7f270c003e70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f271400c210 tx=0x7f270c005830 comp rx=0 tx=0).stop 2024-09-13T11:59:02.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.710+0000 7f272a8b8640 1 -- 172.21.15.100:0/47645229 shutdown_connections 2024-09-13T11:59:02.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.710+0000 7f272a8b8640 1 --2- 172.21.15.100:0/47645229 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f270c003a50 0x7f270c003e70 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.710+0000 7f272a8b8640 1 -- 172.21.15.100:0/47645229 >> 172.21.15.100:0/47645229 conn(0x7f27240a83f0 msgr2=0x7f27240a48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:02.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.710+0000 7f272a8b8640 1 -- 172.21.15.100:0/47645229 shutdown_connections 2024-09-13T11:59:02.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.710+0000 7f272a8b8640 1 -- 172.21.15.100:0/47645229 wait complete. 2024-09-13T11:59:02.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.711+0000 7f272a8b8640 10 monclient: build_initial_monmap 2024-09-13T11:59:02.731 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.711+0000 7f272a8b8640 10 monclient: monmap: 2024-09-13T11:59:02.731 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:02.732 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:02.732 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:02.732 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:02.732 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:02.732 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.732 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:02.732 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:02.732 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:02.732 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.732 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.711+0000 7f272a8b8640 1 Processor -- start 2024-09-13T11:59:02.732 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.711+0000 7f272a8b8640 1 -- start start 2024-09-13T11:59:02.732 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.711+0000 7f272a8b8640 10 monclient: init 2024-09-13T11:59:02.732 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.711+0000 7f272a8b8640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:02.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.711+0000 7f272a8b8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:59:02.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.711+0000 7f272a8b8640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f270c003a50 0x7f27241ac530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f272a8b8640 10 monclient(hunting): picked mon.noname-c con 0x7f270c003a50 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:02.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f272a8b8640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27241cc8d0 0x7f27241c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f272a8b8640 10 monclient(hunting): picked mon.noname-a con 0x7f27241cc8d0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:02.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f272a8b8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27241c8eb0 0x7f27241c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f272a8b8640 10 monclient(hunting): picked mon.noname-b con 0x7f27241c8eb0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:02.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f272a8b8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f272a8b8640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f27240a51b0 con 0x7f27241cc8d0 2024-09-13T11:59:02.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f272a8b8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f272a8b8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f27240706b0 con 0x7f27241c8eb0 2024-09-13T11:59:02.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f272a8b8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f272a8b8640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f27240a5330 con 0x7f270c003a50 2024-09-13T11:59:02.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f272a8b8640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:02.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f272a8b8640 10 monclient(hunting): authenticate will time out at 1123.305043s 2024-09-13T11:59:02.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2723fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f270c003a50 0x7f27241ac530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f27237fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27241cc8d0 0x7f27241c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f27237fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27241cc8d0 0x7f27241c63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47084/0 (socket says 172.21.15.100:47084) 2024-09-13T11:59:02.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2723fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f270c003a50 0x7f27241ac530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:51852/0 (socket says 172.21.15.100:51852) 2024-09-13T11:59:02.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f27237fe640 1 -- 172.21.15.100:0/1785792714 learned_addr learned my addr 172.21.15.100:0/1785792714 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:02.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f27237fe640 10 monclient(hunting): get_auth_request con 0x7f27241cc8d0 auth_method 0 2024-09-13T11:59:02.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f27237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f27237fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f27237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2723fff640 10 monclient(hunting): get_auth_request con 0x7f270c003a50 auth_method 0 2024-09-13T11:59:02.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2723fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2723fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2723fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f27237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f27237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f27237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:02.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2723fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2723fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2723fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:02.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2728e55640 1 --2- 172.21.15.100:0/1785792714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27241c8eb0 0x7f27241c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2728e55640 10 monclient(hunting): get_auth_request con 0x7f27241c8eb0 auth_method 0 2024-09-13T11:59:02.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2728e55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2728e55640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2728e55640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f27237fe640 10 monclient(hunting): handle_auth_done global_id 15489 payload 824 2024-09-13T11:59:02.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f27237fe640 1 -- 172.21.15.100:0/1785792714 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f270c003a50 msgr2=0x7f27241ac530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f27237fe640 1 --2- 172.21.15.100:0/1785792714 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f270c003a50 0x7f27241ac530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f27237fe640 1 -- 172.21.15.100:0/1785792714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27241c8eb0 msgr2=0x7f27241c92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f27237fe640 1 --2- 172.21.15.100:0/1785792714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27241c8eb0 0x7f27241c92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f27237fe640 10 monclient: _finish_hunting 0 2024-09-13T11:59:02.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f27237fe640 1 monclient: found mon.noname-a 2024-09-13T11:59:02.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f27237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:02.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f27237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:02.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f27237fe640 1 -- 172.21.15.100:0/1785792714 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f272409f0b0 con 0x7f27241cc8d0 2024-09-13T11:59:02.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2728e55640 1 --2- 172.21.15.100:0/1785792714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27241c8eb0 0x7f27241c92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:59:02.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f27237fe640 1 --2- 172.21.15.100:0/1785792714 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27241cc8d0 0x7f27241c63b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f271800e840 tx=0x7f271800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:02.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2720ff9640 1 -- 172.21.15.100:0/1785792714 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2718017070 con 0x7f27241cc8d0 2024-09-13T11:59:02.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2720ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2720ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:02.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2720ff9640 10 monclient: dump: 2024-09-13T11:59:02.737 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.737 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.737 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.737 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.737 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.737 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.737 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.737 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.737 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.737 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2720ff9640 10 monclient: _finish_auth 0 2024-09-13T11:59:02.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2720ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:59:02.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2720ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:02.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2720ff9640 1 -- 172.21.15.100:0/1785792714 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f271800c940 con 0x7f27241cc8d0 2024-09-13T11:59:02.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f272a8b8640 5 monclient: authenticate success, global_id 15489 2024-09-13T11:59:02.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f272a8b8640 10 monclient: _renew_subs 2024-09-13T11:59:02.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f272a8b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:02.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f272a8b8640 1 -- 172.21.15.100:0/1785792714 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27241acec0 con 0x7f27241cc8d0 2024-09-13T11:59:02.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2720ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:02.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2720ff9640 1 -- 172.21.15.100:0/1785792714 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271800cd50 con 0x7f27241cc8d0 2024-09-13T11:59:02.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2720ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2720ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:02.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f2720ff9640 10 monclient: dump: 2024-09-13T11:59:02.738 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.739 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.739 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.739 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.739 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.739 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.739 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.739 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.739 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.739 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f27227fc640 4 set_mon_vals no callback set 2024-09-13T11:59:02.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f272a8b8640 10 monclient: _renew_subs 2024-09-13T11:59:02.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f272a8b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:02.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.712+0000 7f272a8b8640 1 -- 172.21.15.100:0/1785792714 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27241add30 con 0x7f27241cc8d0 2024-09-13T11:59:02.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.713+0000 7f272a8b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:02.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.713+0000 7f272a8b8640 1 --2- 172.21.15.100:0/1785792714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27241c9830 0x7f272409ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.713+0000 7f272a8b8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.713+0000 7f272a8b8640 1 -- 172.21.15.100:0/1785792714 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f27240a3d60 con 0x7f27241c9830 2024-09-13T11:59:02.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.713+0000 7f2720ff9640 1 -- 172.21.15.100:0/1785792714 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f271801a070 con 0x7f27241cc8d0 2024-09-13T11:59:02.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.713+0000 7f2723fff640 1 --2- 172.21.15.100:0/1785792714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27241c9830 0x7f272409ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.713+0000 7f2723fff640 10 monclient: get_auth_request con 0x7f27241c9830 auth_method 0 2024-09-13T11:59:02.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.713+0000 7f2723fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.713+0000 7f2723fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.713+0000 7f2723fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.713+0000 7f2720ff9640 1 --2- 172.21.15.100:0/1785792714 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f271003dfa0 0x7f2710040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.713+0000 7f2723fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.713+0000 7f2723fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.740 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.713+0000 7f2723fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:02.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.713+0000 7f2720ff9640 1 -- 172.21.15.100:0/1785792714 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7f2718052530 con 0x7f27241cc8d0 2024-09-13T11:59:02.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.714+0000 7f2728e55640 1 --2- 172.21.15.100:0/1785792714 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f271003dfa0 0x7f2710040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.714+0000 7f2728e55640 10 monclient: get_auth_request con 0x7f271003dfa0 auth_method 0 2024-09-13T11:59:02.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.714+0000 7f2723fff640 10 monclient(hunting): handle_auth_done global_id 34117 payload 824 2024-09-13T11:59:02.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.714+0000 7f2723fff640 1 -- 172.21.15.100:0/1785792714 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f27240008d0 con 0x7f27241c9830 2024-09-13T11:59:02.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.714+0000 7f2723fff640 1 --2- 172.21.15.100:0/1785792714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27241c9830 0x7f272409ba00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2714001eb0 tx=0x7f27240704e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:02.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.714+0000 7f2728e55640 1 --2- 172.21.15.100:0/1785792714 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f271003dfa0 0x7f2710040460 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f272406e1e0 tx=0x7f271c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:02.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.714+0000 7f2720ff9640 1 -- 172.21.15.100:0/1785792714 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2714043020 con 0x7f27241c9830 2024-09-13T11:59:02.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.714+0000 7f2720ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.714+0000 7f2720ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:59:02.741 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.714+0000 7f2720ff9640 10 monclient: dump: 2024-09-13T11:59:02.741 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.741 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.742 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.742 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.742 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.742 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.742 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.742 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.742 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.742 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.714+0000 7f2720ff9640 1 -- 172.21.15.100:0/1785792714 <== mon.1 v2:172.21.15.122:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2714038040 con 0x7f27241c9830 2024-09-13T11:59:02.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.714+0000 7f2720ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:02.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.714+0000 7f2720ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:59:02.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.714+0000 7f2720ff9640 1 -- 172.21.15.100:0/1785792714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27241c9830 msgr2=0x7f272409ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.742 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.714+0000 7f2720ff9640 1 --2- 172.21.15.100:0/1785792714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27241c9830 0x7f272409ba00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2714001eb0 tx=0x7f27240704e0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.714+0000 7f2720ff9640 1 -- 172.21.15.100:0/1785792714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27241c9830 msgr2=0x7f272409ba00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:59:02.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.714+0000 7f2720ff9640 1 --2- 172.21.15.100:0/1785792714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27241c9830 0x7f272409ba00 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2714001eb0 tx=0x7f27240704e0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.720+0000 7f272a8b8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:59:02.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.720+0000 7f272a8b8640 1 --2- 172.21.15.100:0/1785792714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f272409bf40 0x7f272409c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.744 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.720+0000 7f272a8b8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.720+0000 7f272a8b8640 1 -- 172.21.15.100:0/1785792714 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f27241c9c10 con 0x7f272409bf40 2024-09-13T11:59:02.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.721+0000 7f2723fff640 1 --2- 172.21.15.100:0/1785792714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f272409bf40 0x7f272409c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.721+0000 7f2723fff640 10 monclient: get_auth_request con 0x7f272409bf40 auth_method 0 2024-09-13T11:59:02.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.721+0000 7f2723fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.721+0000 7f2723fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.721+0000 7f2723fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.721+0000 7f2723fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.721+0000 7f2723fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.721+0000 7f2723fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:02.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.721+0000 7f2723fff640 10 monclient(hunting): handle_auth_done global_id 34120 payload 824 2024-09-13T11:59:02.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.721+0000 7f2723fff640 1 -- 172.21.15.100:0/1785792714 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f27241ad490 con 0x7f272409bf40 2024-09-13T11:59:02.745 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.721+0000 7f2723fff640 1 --2- 172.21.15.100:0/1785792714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f272409bf40 0x7f272409c350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2714000c00 tx=0x7f271402d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:02.746 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.721+0000 7f2720ff9640 1 -- 172.21.15.100:0/1785792714 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2714043040 con 0x7f272409bf40 2024-09-13T11:59:02.746 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.721+0000 7f2720ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.746 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.721+0000 7f2720ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:59:02.746 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.721+0000 7f2720ff9640 10 monclient: dump: 2024-09-13T11:59:02.746 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.746 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.746 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.746 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.746 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.746 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.746 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.746 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.746 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.747 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.747 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.722+0000 7f2720ff9640 1 -- 172.21.15.100:0/1785792714 <== mon.1 v2:172.21.15.122:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f271404a220 con 0x7f272409bf40 2024-09-13T11:59:02.747 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.722+0000 7f2720ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:59:02.747 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.722+0000 7f2720ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:59:02.747 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.722+0000 7f2720ff9640 1 -- 172.21.15.100:0/1785792714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f272409bf40 msgr2=0x7f272409c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.747 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.722+0000 7f2720ff9640 1 --2- 172.21.15.100:0/1785792714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f272409bf40 0x7f272409c350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2714000c00 tx=0x7f271402d000 comp rx=0 tx=0).stop 2024-09-13T11:59:02.747 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.722+0000 7f2720ff9640 1 -- 172.21.15.100:0/1785792714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f272409bf40 msgr2=0x7f272409c350 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:59:02.747 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.722+0000 7f2720ff9640 1 --2- 172.21.15.100:0/1785792714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f272409bf40 0x7f272409c350 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2714000c00 tx=0x7f271402d000 comp rx=0 tx=0).stop 2024-09-13T11:59:02.747 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:59:02.747 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:59:02.748 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:59:02.748 INFO:teuthology.orchestra.run.smithi100.stdout: "state": "peon", 2024-09-13T11:59:02.748 INFO:teuthology.orchestra.run.smithi100.stdout: "election_epoch": 28, 2024-09-13T11:59:02.748 INFO:teuthology.orchestra.run.smithi100.stdout: "uptime": 10824, 2024-09-13T11:59:02.748 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum": [ 2024-09-13T11:59:02.748 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-13T11:59:02.748 INFO:teuthology.orchestra.run.smithi100.stdout: 1, 2024-09-13T11:59:02.748 INFO:teuthology.orchestra.run.smithi100.stdout: 2 2024-09-13T11:59:02.748 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:59:02.748 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_age": 2, 2024-09-13T11:59:02.748 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:59:02.748 INFO:teuthology.orchestra.run.smithi100.stdout: "required_con": "2450521708007866372", 2024-09-13T11:59:02.748 INFO:teuthology.orchestra.run.smithi100.stdout: "required_mon": [ 2024-09-13T11:59:02.749 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:59:02.749 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:59:02.749 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:59:02.749 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:59:02.749 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:59:02.749 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:59:02.749 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:59:02.749 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:59:02.749 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:59:02.749 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:59:02.749 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:59:02.749 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:59:02.749 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:59:02.750 INFO:teuthology.orchestra.run.smithi100.stdout: "quorum_mon": [ 2024-09-13T11:59:02.750 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:59:02.750 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:59:02.750 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:59:02.750 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:59:02.750 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:59:02.750 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:59:02.750 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:59:02.750 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:59:02.750 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:59:02.750 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:59:02.750 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:59:02.750 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:59:02.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.751 INFO:teuthology.orchestra.run.smithi100.stdout: "outside_quorum": [], 2024-09-13T11:59:02.751 INFO:teuthology.orchestra.run.smithi100.stdout: "extra_probe_peers": [], 2024-09-13T11:59:02.751 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_provider": [], 2024-09-13T11:59:02.751 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": { 2024-09-13T11:59:02.751 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-09-13T11:59:02.751 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "0b1f4b4d-fad4-4dd3-86ac-a286d80846c0", 2024-09-13T11:59:02.751 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:59:02.751 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-13T11:52:13.286120Z", 2024-09-13T11:59:02.751 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release": 19, 2024-09-13T11:59:02.751 INFO:teuthology.orchestra.run.smithi100.stdout: "min_mon_release_name": "squid", 2024-09-13T11:59:02.751 INFO:teuthology.orchestra.run.smithi100.stdout: "election_strategy": 1, 2024-09-13T11:59:02.752 INFO:teuthology.orchestra.run.smithi100.stdout: "disallowed_leaders: ": "", 2024-09-13T11:59:02.752 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false, 2024-09-13T11:59:02.752 INFO:teuthology.orchestra.run.smithi100.stdout: "tiebreaker_mon": "", 2024-09-13T11:59:02.752 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_ranks: ": "", 2024-09-13T11:59:02.752 INFO:teuthology.orchestra.run.smithi100.stdout: "features": { 2024-09-13T11:59:02.752 INFO:teuthology.orchestra.run.smithi100.stdout: "persistent": [ 2024-09-13T11:59:02.752 INFO:teuthology.orchestra.run.smithi100.stdout: "kraken", 2024-09-13T11:59:02.752 INFO:teuthology.orchestra.run.smithi100.stdout: "luminous", 2024-09-13T11:59:02.752 INFO:teuthology.orchestra.run.smithi100.stdout: "mimic", 2024-09-13T11:59:02.752 INFO:teuthology.orchestra.run.smithi100.stdout: "osdmap-prune", 2024-09-13T11:59:02.752 INFO:teuthology.orchestra.run.smithi100.stdout: "nautilus", 2024-09-13T11:59:02.753 INFO:teuthology.orchestra.run.smithi100.stdout: "octopus", 2024-09-13T11:59:02.753 INFO:teuthology.orchestra.run.smithi100.stdout: "pacific", 2024-09-13T11:59:02.753 INFO:teuthology.orchestra.run.smithi100.stdout: "elector-pinging", 2024-09-13T11:59:02.753 INFO:teuthology.orchestra.run.smithi100.stdout: "quincy", 2024-09-13T11:59:02.753 INFO:teuthology.orchestra.run.smithi100.stdout: "reef", 2024-09-13T11:59:02.753 INFO:teuthology.orchestra.run.smithi100.stdout: "squid" 2024-09-13T11:59:02.753 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:59:02.753 INFO:teuthology.orchestra.run.smithi100.stdout: "optional": [] 2024-09-13T11:59:02.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.753 INFO:teuthology.orchestra.run.smithi100.stdout: "mons": [ 2024-09-13T11:59:02.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.753 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 0, 2024-09-13T11:59:02.753 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "a", 2024-09-13T11:59:02.753 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:59:02.754 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:59:02.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.754 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:59:02.754 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3300", 2024-09-13T11:59:02.754 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:59:02.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.754 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:59:02.754 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789", 2024-09-13T11:59:02.754 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:59:02.754 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:59:02.754 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:59:02.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.754 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6789/0", 2024-09-13T11:59:02.755 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-13T11:59:02.755 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:59:02.755 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:59:02.755 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:59:02.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.755 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 1, 2024-09-13T11:59:02.755 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "b", 2024-09-13T11:59:02.755 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:59:02.755 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:59:02.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.755 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:59:02.755 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:3300", 2024-09-13T11:59:02.756 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:59:02.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.756 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:59:02.756 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789", 2024-09-13T11:59:02.756 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:59:02.756 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:59:02.756 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:59:02.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.756 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.122:6789/0", 2024-09-13T11:59:02.756 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.122:6789/0", 2024-09-13T11:59:02.756 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:59:02.756 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:59:02.757 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:59:02.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.757 INFO:teuthology.orchestra.run.smithi100.stdout: "rank": 2, 2024-09-13T11:59:02.757 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "c", 2024-09-13T11:59:02.757 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-13T11:59:02.757 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-13T11:59:02.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.757 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-13T11:59:02.757 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:3301", 2024-09-13T11:59:02.757 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:59:02.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.757 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-09-13T11:59:02.758 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790", 2024-09-13T11:59:02.758 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 0 2024-09-13T11:59:02.758 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:59:02.758 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:59:02.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.758 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6790/0", 2024-09-13T11:59:02.758 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-13T11:59:02.758 INFO:teuthology.orchestra.run.smithi100.stdout: "priority": 0, 2024-09-13T11:59:02.758 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-13T11:59:02.758 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "{}" 2024-09-13T11:59:02.758 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:59:02.758 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:59:02.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.759 INFO:teuthology.orchestra.run.smithi100.stdout: "feature_map": { 2024-09-13T11:59:02.759 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": [ 2024-09-13T11:59:02.759 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.759 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:59:02.759 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:59:02.759 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:59:02.759 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:59:02.759 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-13T11:59:02.759 INFO:teuthology.orchestra.run.smithi100.stdout: "client": [ 2024-09-13T11:59:02.759 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T11:59:02.759 INFO:teuthology.orchestra.run.smithi100.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:59:02.759 INFO:teuthology.orchestra.run.smithi100.stdout: "release": "squid", 2024-09-13T11:59:02.759 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 1 2024-09-13T11:59:02.759 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T11:59:02.760 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T11:59:02.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T11:59:02.760 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": false 2024-09-13T11:59:02.760 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:59:02.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.722+0000 7f272a8b8640 1 -- 172.21.15.100:0/1785792714 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f271003dfa0 msgr2=0x7f2710040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.722+0000 7f272a8b8640 1 --2- 172.21.15.100:0/1785792714 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f271003dfa0 0x7f2710040460 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f272406e1e0 tx=0x7f271c009040 comp rx=0 tx=0).stop 2024-09-13T11:59:02.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.722+0000 7f272a8b8640 10 monclient: shutdown 2024-09-13T11:59:02.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.722+0000 7f272a8b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:02.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.722+0000 7f272a8b8640 1 -- 172.21.15.100:0/1785792714 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27241cc8d0 msgr2=0x7f27241c63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.722+0000 7f272a8b8640 1 --2- 172.21.15.100:0/1785792714 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27241cc8d0 0x7f27241c63b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f271800e840 tx=0x7f271800ed00 comp rx=0 tx=0).stop 2024-09-13T11:59:02.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.722+0000 7f2728e55640 1 -- 172.21.15.100:0/1785792714 reap_dead start 2024-09-13T11:59:02.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.722+0000 7f272a8b8640 1 -- 172.21.15.100:0/1785792714 shutdown_connections 2024-09-13T11:59:02.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.722+0000 7f272a8b8640 1 --2- 172.21.15.100:0/1785792714 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27241cc8d0 0x7f27241c63b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.722+0000 7f272a8b8640 1 -- 172.21.15.100:0/1785792714 >> 172.21.15.100:0/1785792714 conn(0x7f27240a83f0 msgr2=0x7f2724072f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:02.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.722+0000 7f272a8b8640 1 -- 172.21.15.100:0/1785792714 shutdown_connections 2024-09-13T11:59:02.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.722+0000 7f272a8b8640 1 -- 172.21.15.100:0/1785792714 wait complete. 2024-09-13T11:59:02.762 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T11:59:02.762 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-13T11:59:02.762 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-13T11:59:02.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.858+0000 7f3d96471640 10 monclient: get_monmap_and_config 2024-09-13T11:59:02.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.858+0000 7f3d96471640 10 monclient: build_initial_monmap 2024-09-13T11:59:02.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.858+0000 7f3d96471640 10 monclient: monmap: 2024-09-13T11:59:02.860 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:02.860 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.860 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:02.859330+0000 2024-09-13T11:59:02.860 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:02.859330+0000 2024-09-13T11:59:02.860 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:02.860 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.860 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:02.860 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:02.860 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:02.860 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:02.860 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:02.860 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.859+0000 7f3d96471640 1 Processor -- start 2024-09-13T11:59:02.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.859+0000 7f3d96471640 1 -- start start 2024-09-13T11:59:02.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.859+0000 7f3d96471640 10 monclient: init 2024-09-13T11:59:02.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.859+0000 7f3d96471640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:02.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.859+0000 7f3d96471640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-13T11:59:02.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.859+0000 7f3d96471640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d900a2640 0x7f3d900a2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.859+0000 7f3d96471640 10 monclient(hunting): picked mon.noname-b con 0x7f3d900a2640 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:02.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.859+0000 7f3d96471640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3d900a2f60 addr v1:172.21.15.100:6789/0 2024-09-13T11:59:02.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.859+0000 7f3d96471640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3d90135b10 0x7f3d9013d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.859+0000 7f3d96471640 10 monclient(hunting): picked mon.noname-c con 0x7f3d90135b10 addr v2:172.21.15.122:3300/0 2024-09-13T11:59:02.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.859+0000 7f3d96471640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d90076830 con 0x7f3d900a2f60 2024-09-13T11:59:02.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.859+0000 7f3d96471640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.859+0000 7f3d96471640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f3d90076f50 con 0x7f3d90135b10 2024-09-13T11:59:02.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.859+0000 7f3d96471640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.859+0000 7f3d96471640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f3d9009f040 con 0x7f3d900a2640 2024-09-13T11:59:02.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.859+0000 7f3d96471640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:02.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.859+0000 7f3d96471640 10 monclient(hunting): authenticate will time out at 1123.452930s 2024-09-13T11:59:02.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.859+0000 7f3d8ffff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d900a2640 0x7f3d900a2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d8f7fe640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f3d900a2f60 0x7f3d90135400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:57066/0 (socket says 172.21.15.100:57066) 2024-09-13T11:59:02.869 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d8f7fe640 1 -- 172.21.15.100:0/2777375120 learned_addr learned my addr 172.21.15.100:0/2777375120 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:59:02.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d8ffff640 10 monclient(hunting): get_auth_request con 0x7f3d900a2640 auth_method 0 2024-09-13T11:59:02.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:02.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d94a0e640 1 --2- 172.21.15.100:0/2777375120 >> v2:172.21.15.122:3300/0 conn(0x7f3d90135b10 0x7f3d9013d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d94a0e640 10 monclient(hunting): get_auth_request con 0x7f3d90135b10 auth_method 0 2024-09-13T11:59:02.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d94a0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d94a0e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d94a0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d8ffff640 10 monclient(hunting): handle_auth_done global_id 24587 payload 274 2024-09-13T11:59:02.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d8effd640 1 -- 172.21.15.100:0/2777375120 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2688413024 0 0) 0x7f3d90076830 con 0x7f3d900a2f60 2024-09-13T11:59:02.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d8ffff640 1 -- 172.21.15.100:0/2777375120 >> v2:172.21.15.122:3300/0 conn(0x7f3d90135b10 msgr2=0x7f3d9013d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:02.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d8ffff640 1 --2- 172.21.15.100:0/2777375120 >> v2:172.21.15.122:3300/0 conn(0x7f3d90135b10 0x7f3d9013d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d8ffff640 1 -- 172.21.15.100:0/2777375120 >> v1:172.21.15.100:6789/0 conn(0x7f3d900a2f60 legacy=0x7f3d90135400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d8ffff640 10 monclient: _finish_hunting 0 2024-09-13T11:59:02.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d8ffff640 1 monclient: found mon.noname-b 2024-09-13T11:59:02.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:02.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d8ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:59:02.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d8ffff640 1 -- 172.21.15.100:0/2777375120 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d900765b0 con 0x7f3d900a2640 2024-09-13T11:59:02.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d8effd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:59:02.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.860+0000 7f3d8ffff640 1 --2- 172.21.15.100:0/2777375120 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d900a2640 0x7f3d900a2a20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3d800097a0 tx=0x7f3d8002a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af8f655d2054a527 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:02.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d8effd640 1 -- 172.21.15.100:0/2777375120 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d80037070 con 0x7f3d900a2640 2024-09-13T11:59:02.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d8effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:59:02.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d8effd640 10 monclient: dump: 2024-09-13T11:59:02.871 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.872 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.872 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.872 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.872 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.872 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.872 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.872 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.872 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.872 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.872 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d8effd640 10 monclient: _finish_auth 0 2024-09-13T11:59:02.872 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d8effd640 10 monclient: _check_auth_tickets 2024-09-13T11:59:02.872 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:02.872 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d8effd640 1 -- 172.21.15.100:0/2777375120 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d8002adb0 con 0x7f3d900a2640 2024-09-13T11:59:02.872 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d96471640 5 monclient: authenticate success, global_id 24587 2024-09-13T11:59:02.872 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d96471640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:02.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d8effd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:02.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d96471640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:02.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d96471640 4 set_mon_vals no callback set 2024-09-13T11:59:02.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d8effd640 1 -- 172.21.15.100:0/2777375120 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d80032660 con 0x7f3d900a2640 2024-09-13T11:59:02.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d8effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:59:02.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d8effd640 10 monclient: dump: 2024-09-13T11:59:02.873 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.873 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.873 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.873 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.873 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.873 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.874 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.874 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.874 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.874 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d96471640 10 monclient: shutdown 2024-09-13T11:59:02.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d96471640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:02.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d96471640 1 -- 172.21.15.100:0/2777375120 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d900a2640 msgr2=0x7f3d900a2a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d96471640 1 --2- 172.21.15.100:0/2777375120 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d900a2640 0x7f3d900a2a20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3d800097a0 tx=0x7f3d8002a060 comp rx=0 tx=0).stop 2024-09-13T11:59:02.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d96471640 1 -- 172.21.15.100:0/2777375120 shutdown_connections 2024-09-13T11:59:02.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d96471640 1 --2- 172.21.15.100:0/2777375120 >> v2:172.21.15.122:3300/0 conn(0x7f3d90135b10 0x7f3d9013d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d96471640 1 --2- 172.21.15.100:0/2777375120 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d900a2640 0x7f3d900a2a20 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d96471640 1 -- 172.21.15.100:0/2777375120 >> 172.21.15.100:0/2777375120 conn(0x7f3d900a7260 msgr2=0x7f3d900a3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:02.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d96471640 1 -- 172.21.15.100:0/2777375120 shutdown_connections 2024-09-13T11:59:02.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.861+0000 7f3d96471640 1 -- 172.21.15.100:0/2777375120 wait complete. 2024-09-13T11:59:02.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.862+0000 7f3d96471640 10 monclient: build_initial_monmap 2024-09-13T11:59:02.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.862+0000 7f3d96471640 10 monclient: monmap: 2024-09-13T11:59:02.875 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:02.875 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:02.875 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:02.875 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:02.875 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:02.875 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.875 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:02.875 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:02.875 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:02.875 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.862+0000 7f3d96471640 1 Processor -- start 2024-09-13T11:59:02.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.862+0000 7f3d96471640 1 -- start start 2024-09-13T11:59:02.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.862+0000 7f3d96471640 10 monclient: init 2024-09-13T11:59:02.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d96471640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:02.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d96471640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:59:02.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d96471640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d900a2640 0x7f3d901ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d96471640 10 monclient(hunting): picked mon.noname-c con 0x7f3d900a2640 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:02.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d96471640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d900a2f60 0x7f3d901cade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d96471640 10 monclient(hunting): picked mon.noname-b con 0x7f3d900a2f60 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:02.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d96471640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d90135b10 0x7f3d90121d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d96471640 10 monclient(hunting): picked mon.noname-a con 0x7f3d90135b10 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:02.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d96471640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d96471640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f3d900a3340 con 0x7f3d90135b10 2024-09-13T11:59:02.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d96471640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d96471640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f3d900a0fe0 con 0x7f3d900a2f60 2024-09-13T11:59:02.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d96471640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:02.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d96471640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f3d90073c80 con 0x7f3d900a2640 2024-09-13T11:59:02.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d96471640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:02.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d96471640 10 monclient(hunting): authenticate will time out at 1123.456176s 2024-09-13T11:59:02.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8ffff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d900a2640 0x7f3d901ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d94a0e640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d90135b10 0x7f3d90121d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8ffff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d900a2640 0x7f3d901ca8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:51870/0 (socket says 172.21.15.100:51870) 2024-09-13T11:59:02.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8ffff640 1 -- 172.21.15.100:0/2394083112 learned_addr learned my addr 172.21.15.100:0/2394083112 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:02.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8ffff640 10 monclient(hunting): get_auth_request con 0x7f3d900a2640 auth_method 0 2024-09-13T11:59:02.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d94a0e640 10 monclient(hunting): get_auth_request con 0x7f3d90135b10 auth_method 0 2024-09-13T11:59:02.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d94a0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d94a0e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d94a0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.878 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.878 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8f7fe640 1 --2- 172.21.15.100:0/2394083112 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d900a2f60 0x7f3d901cade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.878 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:02.878 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d94a0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.878 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d94a0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.878 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d94a0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:02.878 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8f7fe640 10 monclient(hunting): get_auth_request con 0x7f3d900a2f60 auth_method 0 2024-09-13T11:59:02.878 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:02.878 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:02.878 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:02.878 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:02.878 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:02.878 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:02.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d94a0e640 10 monclient(hunting): handle_auth_done global_id 15495 payload 824 2024-09-13T11:59:02.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d94a0e640 1 -- 172.21.15.100:0/2394083112 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d900a2640 msgr2=0x7f3d901ca8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d94a0e640 1 --2- 172.21.15.100:0/2394083112 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d900a2640 0x7f3d901ca8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d94a0e640 1 -- 172.21.15.100:0/2394083112 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d900a2f60 msgr2=0x7f3d901cade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:02.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d94a0e640 1 --2- 172.21.15.100:0/2394083112 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d900a2f60 0x7f3d901cade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:02.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d94a0e640 10 monclient: _finish_hunting 0 2024-09-13T11:59:02.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d94a0e640 1 monclient: found mon.noname-a 2024-09-13T11:59:02.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d94a0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:02.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d94a0e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:02.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d94a0e640 1 -- 172.21.15.100:0/2394083112 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d90125a60 con 0x7f3d90135b10 2024-09-13T11:59:02.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8ffff640 1 --2- 172.21.15.100:0/2394083112 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d900a2640 0x7f3d901ca8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:59:02.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d94a0e640 1 --2- 172.21.15.100:0/2394083112 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d90135b10 0x7f3d90121d60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3d88007d90 tx=0x7f3d8800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:02.879 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8cff9640 1 -- 172.21.15.100:0/2394083112 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d8800edd0 con 0x7f3d90135b10 2024-09-13T11:59:02.880 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.880 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:02.880 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8cff9640 10 monclient: dump: 2024-09-13T11:59:02.880 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.880 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.880 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.880 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.880 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.880 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.880 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.880 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.880 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.880 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.880 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8cff9640 10 monclient: _finish_auth 0 2024-09-13T11:59:02.881 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8cff9640 10 monclient: _check_auth_tickets 2024-09-13T11:59:02.881 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:02.881 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8cff9640 1 -- 172.21.15.100:0/2394083112 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d88003990 con 0x7f3d90135b10 2024-09-13T11:59:02.881 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d96471640 5 monclient: authenticate success, global_id 15495 2024-09-13T11:59:02.881 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8cff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:02.881 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d96471640 10 monclient: _renew_subs 2024-09-13T11:59:02.881 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8cff9640 1 -- 172.21.15.100:0/2394083112 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d88003d80 con 0x7f3d90135b10 2024-09-13T11:59:02.881 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d96471640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:02.881 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d96471640 1 -- 172.21.15.100:0/2394083112 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d90074f90 con 0x7f3d90135b10 2024-09-13T11:59:02.881 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:02.881 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:02.881 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8cff9640 10 monclient: dump: 2024-09-13T11:59:02.881 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:02.882 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:02.882 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.882 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:02.882 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:02.882 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:02.882 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:02.882 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:02.882 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:02.882 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:02.882 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.863+0000 7f3d8e7fc640 4 set_mon_vals no callback set 2024-09-13T11:59:02.882 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.864+0000 7f3d96471640 10 monclient: _renew_subs 2024-09-13T11:59:02.882 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.864+0000 7f3d96471640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:02.882 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.864+0000 7f3d96471640 1 -- 172.21.15.100:0/2394083112 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d90125780 con 0x7f3d90135b10 2024-09-13T11:59:02.882 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.864+0000 7f3d8cff9640 1 -- 172.21.15.100:0/2394083112 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f3d8801b020 con 0x7f3d90135b10 2024-09-13T11:59:02.882 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.864+0000 7f3d96471640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:02.883 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.864+0000 7f3d96471640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:02.883 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.864+0000 7f3d96471640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:02.883 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.864+0000 7f3d8cff9640 1 --2- 172.21.15.100:0/2394083112 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f3d7c03dfa0 0x7f3d7c040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:02.883 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.864+0000 7f3d96471640 1 -- 172.21.15.100:0/2394083112 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d900a4100 con 0x7f3d90135b10 2024-09-13T11:59:02.883 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.865+0000 7f3d8cff9640 1 -- 172.21.15.100:0/2394083112 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7f3d880526d0 con 0x7f3d90135b10 2024-09-13T11:59:02.883 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.865+0000 7f3d8ffff640 1 --2- 172.21.15.100:0/2394083112 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f3d7c03dfa0 0x7f3d7c040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:02.883 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.865+0000 7f3d8ffff640 10 monclient: get_auth_request con 0x7f3d7c03dfa0 auth_method 0 2024-09-13T11:59:02.883 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.865+0000 7f3d8ffff640 1 --2- 172.21.15.100:0/2394083112 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f3d7c03dfa0 0x7f3d7c040460 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3d8002a0a0 tx=0x7f3d8002e000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:02.883 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.869+0000 7f3d8cff9640 1 -- 172.21.15.100:0/2394083112 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f3d8800a040 con 0x7f3d90135b10 2024-09-13T11:59:02.884 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.869+0000 7f3d8cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:02.884 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:02.869+0000 7f3d8cff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:59:03.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:03.109+0000 7f3d96471640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-13T11:59:03.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:03.109+0000 7f3d96471640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-13T11:59:03.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:03.109+0000 7f3d96471640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:03.111 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:03.109+0000 7f3d96471640 1 -- 172.21.15.100:0/2394083112 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f3d900771e0 con 0x7f3d90135b10 2024-09-13T11:59:03.111 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:03.110+0000 7f3d8cff9640 1 -- 172.21.15.100:0/2394083112 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f3d880523a0 con 0x7f3d90135b10 2024-09-13T11:59:03.111 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:03.110+0000 7f3d8cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-13T11:59:03.111 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:03.110+0000 7f3d8cff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:59:03.112 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.110+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.112 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.110+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=11,kv=13,logm=64} crc {auth=1356782785,config=1097430493,health=1096256309,kv=4190740400,logm=3552132744}) 2024-09-13T11:59:03.112 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.110+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=11,kv=13,logm=64} crc {auth=1356782785,config=1097430493,health=1993131081,kv=4190740400,logm=608075974}) 2024-09-13T11:59:03.112 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.110+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.112 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.110+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=11,kv=13,logm=64} crc {auth=1356782785,config=1097430493,health=1096256309,kv=4190740400,logm=3552132744}) 2024-09-13T11:59:03.113 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.110+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=11,kv=10,logm=67} crc {auth=1599114015,config=1097430493,health=1791238353,kv=4150625761,logm=3594824345}) 2024-09-13T11:59:03.113 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.111+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.113 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.111+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3287348587}) 2024-09-13T11:59:03.113 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.111+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=810386373}) 2024-09-13T11:59:03.113 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.111+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.113 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.111+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3287348587}) 2024-09-13T11:59:03.113 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.111+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3855698521}) 2024-09-13T11:59:03.114 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.112+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.114 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.112+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1748481434}) 2024-09-13T11:59:03.114 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.112+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1747090701}) 2024-09-13T11:59:03.114 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.112+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.114 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.112+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1748481434}) 2024-09-13T11:59:03.114 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.112+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1826683266}) 2024-09-13T11:59:03.115 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:03.112+0000 7f3d96471640 1 -- 172.21.15.100:0/2394083112 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f3d7c03dfa0 msgr2=0x7f3d7c040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:03.115 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:03.112+0000 7f3d96471640 1 --2- 172.21.15.100:0/2394083112 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f3d7c03dfa0 0x7f3d7c040460 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3d8002a0a0 tx=0x7f3d8002e000 comp rx=0 tx=0).stop 2024-09-13T11:59:03.115 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:03.112+0000 7f3d96471640 10 monclient: shutdown 2024-09-13T11:59:03.115 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:03.112+0000 7f3d96471640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:03.115 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:03.112+0000 7f3d96471640 1 -- 172.21.15.100:0/2394083112 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d90135b10 msgr2=0x7f3d90121d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:03.115 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:03.112+0000 7f3d96471640 1 --2- 172.21.15.100:0/2394083112 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d90135b10 0x7f3d90121d60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3d88007d90 tx=0x7f3d8800e4b0 comp rx=0 tx=0).stop 2024-09-13T11:59:03.116 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:03.112+0000 7f3d96471640 1 -- 172.21.15.100:0/2394083112 shutdown_connections 2024-09-13T11:59:03.116 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:03.112+0000 7f3d96471640 1 --2- 172.21.15.100:0/2394083112 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f3d7c03dfa0 0x7f3d7c040460 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:03.116 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:03.112+0000 7f3d96471640 1 --2- 172.21.15.100:0/2394083112 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d90135b10 0x7f3d90121d60 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:03.116 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:03.112+0000 7f3d96471640 1 --2- 172.21.15.100:0/2394083112 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d900a2f60 0x7f3d901cade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:03.116 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:03.112+0000 7f3d96471640 1 --2- 172.21.15.100:0/2394083112 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d900a2640 0x7f3d901ca8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:03.116 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:03.112+0000 7f3d96471640 1 -- 172.21.15.100:0/2394083112 >> 172.21.15.100:0/2394083112 conn(0x7f3d900a7260 msgr2=0x7f3d90071ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:03.116 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:03.112+0000 7f3d96471640 1 -- 172.21.15.100:0/2394083112 shutdown_connections 2024-09-13T11:59:03.116 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:03.112+0000 7f3d96471640 1 -- 172.21.15.100:0/2394083112 wait complete. 2024-09-13T11:59:03.119 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.113+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.119 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.113+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1965332724}) 2024-09-13T11:59:03.119 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.113+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3725394175}) 2024-09-13T11:59:03.119 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.113+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.120 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.113+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1965332724}) 2024-09-13T11:59:03.120 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.113+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3818896045}) 2024-09-13T11:59:03.120 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.113+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.120 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.113+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4111725722}) 2024-09-13T11:59:03.120 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.113+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4021734063}) 2024-09-13T11:59:03.120 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.113+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.120 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.113+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4111725722}) 2024-09-13T11:59:03.120 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.113+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3035019703}) 2024-09-13T11:59:03.120 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.114+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.120 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.114+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=513507288}) 2024-09-13T11:59:03.120 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.114+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2236602339}) 2024-09-13T11:59:03.120 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.114+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.121 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.114+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=513507288}) 2024-09-13T11:59:03.121 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.114+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2478784450}) 2024-09-13T11:59:03.121 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.115+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.121 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.115+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=817110569}) 2024-09-13T11:59:03.121 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.115+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2902392096}) 2024-09-13T11:59:03.121 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.115+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.121 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.115+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=817110569}) 2024-09-13T11:59:03.121 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.115+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1084368856}) 2024-09-13T11:59:03.121 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.116+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.121 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.116+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062303080}) 2024-09-13T11:59:03.121 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.116+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=163156687}) 2024-09-13T11:59:03.121 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.116+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.121 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.116+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062303080}) 2024-09-13T11:59:03.122 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.116+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2024418087}) 2024-09-13T11:59:03.122 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.116+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.122 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.116+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2845935926}) 2024-09-13T11:59:03.122 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.116+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2989166008}) 2024-09-13T11:59:03.122 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.116+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.122 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.116+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2845935926}) 2024-09-13T11:59:03.122 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.116+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1010259832}) 2024-09-13T11:59:03.122 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.117+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.122 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.117+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1599775893}) 2024-09-13T11:59:03.122 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.117+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=849030926}) 2024-09-13T11:59:03.122 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.117+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.122 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.117+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1599775893}) 2024-09-13T11:59:03.122 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.117+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1208261767}) 2024-09-13T11:59:03.123 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.118+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.123 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.118+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1045095801}) 2024-09-13T11:59:03.123 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.118+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3328944872}) 2024-09-13T11:59:03.123 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.118+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.123 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.118+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1045095801}) 2024-09-13T11:59:03.123 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.118+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3567023696}) 2024-09-13T11:59:03.125 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.119+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.125 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.119+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2587053709}) 2024-09-13T11:59:03.125 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.119+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=364016322}) 2024-09-13T11:59:03.125 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.119+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.125 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.119+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2587053709}) 2024-09-13T11:59:03.125 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.119+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1613769229}) 2024-09-13T11:59:03.125 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.119+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.126 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.119+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2712927340}) 2024-09-13T11:59:03.126 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.119+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2037003696}) 2024-09-13T11:59:03.126 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.119+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.126 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.119+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2712927340}) 2024-09-13T11:59:03.126 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.119+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3255032396}) 2024-09-13T11:59:03.126 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.120+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.126 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.120+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4005291261}) 2024-09-13T11:59:03.126 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.120+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3349285480}) 2024-09-13T11:59:03.126 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.120+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.126 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.120+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4005291261}) 2024-09-13T11:59:03.126 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.120+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1535082381}) 2024-09-13T11:59:03.126 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.121+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.127 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.121+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=23,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=20,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=20} crc {logm=2616531464,mdsmap=3458900325,mgr=4016471625,mgr_command_descs=3192186153,mgr_metadata=4154780615,mgrstat=1796390544,mon_config_key=872455282,monmap=157995306,nvmeofgw=598337686,osd_metadata=789425444,osd_pg_creating=189829850,osd_snap=841190635,osdmap=1171696912}) 2024-09-13T11:59:03.127 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.121+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=22,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=9,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osdmap=28} crc {logm=2564496992,mdsmap=3458900325,mgr=3464231972,mgr_command_descs=3192186153,mgrstat=84967008,mon_config_key=405219354,monmap=157995306,nvmeofgw=598337686,osd_metadata=789425444,osd_pg_creating=189829850,osdmap=841280972}) 2024-09-13T11:59:03.127 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.121+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.127 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.121+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=23,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=20,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=20} crc {logm=2616531464,mdsmap=3458900325,mgr=4016471625,mgr_command_descs=3192186153,mgr_metadata=4154780615,mgrstat=1796390544,mon_config_key=872455282,monmap=157995306,nvmeofgw=598337686,osd_metadata=789425444,osd_pg_creating=189829850,osd_snap=841190635,osdmap=1171696912}) 2024-09-13T11:59:03.127 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.121+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=23,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=18,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=24} crc {logm=2033455111,mdsmap=3458900325,mgr=2459773887,mgr_command_descs=3192186153,mgr_metadata=4154780615,mgrstat=49956641,mon_config_key=466419833,monmap=157995306,nvmeofgw=598337686,osd_metadata=789425444,osd_pg_creating=189829850,osd_snap=3058936622,osdmap=125630581}) 2024-09-13T11:59:03.127 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.122+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.127 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.122+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1882966523}) 2024-09-13T11:59:03.127 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.122+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4087474527}) 2024-09-13T11:59:03.127 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.122+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.127 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.122+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1882966523}) 2024-09-13T11:59:03.127 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.122+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=943051290}) 2024-09-13T11:59:03.127 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.122+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.127 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.122+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=600115482}) 2024-09-13T11:59:03.128 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.122+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1154790351}) 2024-09-13T11:59:03.128 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.122+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.128 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.122+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=600115482}) 2024-09-13T11:59:03.128 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.122+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4187237181}) 2024-09-13T11:59:03.129 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-13T11:59:03.130 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.123+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.130 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.123+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=368639756}) 2024-09-13T11:59:03.130 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.123+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=943751515}) 2024-09-13T11:59:03.130 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.123+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.130 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.123+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=368639756}) 2024-09-13T11:59:03.130 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.123+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4059965409}) 2024-09-13T11:59:03.130 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.125+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.130 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.125+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3055796325}) 2024-09-13T11:59:03.130 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.125+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1398387686}) 2024-09-13T11:59:03.130 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.125+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.130 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.125+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3055796325}) 2024-09-13T11:59:03.130 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.125+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3084250437}) 2024-09-13T11:59:03.131 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.130+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.131 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.130+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=41} crc {osdmap=2416550711}) 2024-09-13T11:59:03.131 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.130+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=39} crc {osdmap=39724669}) 2024-09-13T11:59:03.131 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.130+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:59:03.131 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.130+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=41} crc {osdmap=2416550711}) 2024-09-13T11:59:03.132 INFO:tasks.ceph.mon.a.smithi100.stderr:2024-09-13T11:59:03.130+0000 7fb50b90e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=42} crc {osdmap=2552166758}) 2024-09-13T11:59:04.129 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-13T11:59:04.130 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:59:04.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.235+0000 7f125cdab640 10 monclient: get_monmap_and_config 2024-09-13T11:59:04.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.235+0000 7f125cdab640 10 monclient: build_initial_monmap 2024-09-13T11:59:04.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.235+0000 7f125cdab640 10 monclient: monmap: 2024-09-13T11:59:04.238 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:04.238 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:04.238 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:04.237603+0000 2024-09-13T11:59:04.238 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:04.237603+0000 2024-09-13T11:59:04.238 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:04.238 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:04.238 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:04.238 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:04.238 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:04.239 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:04.239 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:04.239 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:04.247 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.236+0000 7f125cdab640 1 Processor -- start 2024-09-13T11:59:04.247 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.236+0000 7f125cdab640 1 -- start start 2024-09-13T11:59:04.247 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.236+0000 7f125cdab640 10 monclient: init 2024-09-13T11:59:04.247 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f125cdab640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:04.247 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f125cdab640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-13T11:59:04.247 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f125cdab640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f12580a15a0 0x7f12580a1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:04.247 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f125cdab640 10 monclient(hunting): picked mon.noname-c con 0x7f12580a15a0 addr v2:172.21.15.122:3300/0 2024-09-13T11:59:04.247 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f125cdab640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f125809f6d0 addr v1:172.21.15.100:6789/0 2024-09-13T11:59:04.247 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f125cdab640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12580a1ec0 0x7f1258134ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:04.247 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f125cdab640 10 monclient(hunting): picked mon.noname-b con 0x7f12580a1ec0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:04.247 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f125cdab640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f125809ecf0 con 0x7f125809f6d0 2024-09-13T11:59:04.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f125cdab640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:04.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f125cdab640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f125809d3b0 con 0x7f12580a15a0 2024-09-13T11:59:04.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f125cdab640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:04.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f125cdab640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f125809ea10 con 0x7f12580a1ec0 2024-09-13T11:59:04.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f125cdab640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:04.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f125cdab640 10 monclient(hunting): authenticate will time out at 1124.831227s 2024-09-13T11:59:04.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f1256d9d640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12580a1ec0 0x7f1258134ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:04.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f1256d9d640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12580a1ec0 0x7f1258134ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:51872/0 (socket says 172.21.15.100:51872) 2024-09-13T11:59:04.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f1255d9b640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f125809f6d0 0x7f12581343d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:57068/0 (socket says 172.21.15.100:57068) 2024-09-13T11:59:04.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f1256d9d640 1 -- 172.21.15.100:0/2917699038 learned_addr learned my addr 172.21.15.100:0/2917699038 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:04.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f1256d9d640 10 monclient(hunting): get_auth_request con 0x7f12580a1ec0 auth_method 0 2024-09-13T11:59:04.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f1256d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:04.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f1256d9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:04.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f1256d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:04.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f1256d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:04.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f1256d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:04.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f1256d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:04.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f125659c640 1 --2- 172.21.15.100:0/2917699038 >> v2:172.21.15.122:3300/0 conn(0x7f12580a15a0 0x7f12580a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:04.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f125659c640 10 monclient(hunting): get_auth_request con 0x7f12580a15a0 auth_method 0 2024-09-13T11:59:04.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f125659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:04.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f125659c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:04.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f125659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:04.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f1256d9d640 10 monclient(hunting): handle_auth_done global_id 24593 payload 274 2024-09-13T11:59:04.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f1256d9d640 1 -- 172.21.15.100:0/2917699038 >> v2:172.21.15.122:3300/0 conn(0x7f12580a15a0 msgr2=0x7f12580a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:04.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f1256d9d640 1 --2- 172.21.15.100:0/2917699038 >> v2:172.21.15.122:3300/0 conn(0x7f12580a15a0 0x7f12580a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f1256d9d640 1 -- 172.21.15.100:0/2917699038 >> v1:172.21.15.100:6789/0 conn(0x7f125809f6d0 legacy=0x7f12581343d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:04.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f1256d9d640 10 monclient: _finish_hunting 0 2024-09-13T11:59:04.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f1256d9d640 1 monclient: found mon.noname-b 2024-09-13T11:59:04.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f1256d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:04.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f1256d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:59:04.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f125559a640 1 -- 172.21.15.100:0/2917699038 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3561237889 0 0) 0x7f125809ecf0 con 0x7f125809f6d0 2024-09-13T11:59:04.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f1256d9d640 1 -- 172.21.15.100:0/2917699038 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12580a2300 con 0x7f12580a1ec0 2024-09-13T11:59:04.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.237+0000 7f125559a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:59:04.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.238+0000 7f1256d9d640 1 --2- 172.21.15.100:0/2917699038 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12580a1ec0 0x7f1258134ae0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1250009e50 tx=0x7f125002a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75656166d0279d70 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:04.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.238+0000 7f125559a640 1 -- 172.21.15.100:0/2917699038 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1250036070 con 0x7f12580a1ec0 2024-09-13T11:59:04.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.238+0000 7f125559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:04.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.238+0000 7f125559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:59:04.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.238+0000 7f125559a640 10 monclient: dump: 2024-09-13T11:59:04.251 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:04.251 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:04.251 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:04.251 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:04.251 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:04.251 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:04.251 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:04.251 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:04.251 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:04.251 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:04.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.238+0000 7f125559a640 10 monclient: _finish_auth 0 2024-09-13T11:59:04.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.238+0000 7f125559a640 10 monclient: _check_auth_tickets 2024-09-13T11:59:04.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.238+0000 7f125559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:04.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.238+0000 7f125559a640 1 -- 172.21.15.100:0/2917699038 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1250031600 con 0x7f12580a1ec0 2024-09-13T11:59:04.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.238+0000 7f125cdab640 5 monclient: authenticate success, global_id 24593 2024-09-13T11:59:04.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.238+0000 7f125cdab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:04.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.238+0000 7f125559a640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:04.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.238+0000 7f125cdab640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:04.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.238+0000 7f125559a640 1 -- 172.21.15.100:0/2917699038 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12500319d0 con 0x7f12580a1ec0 2024-09-13T11:59:04.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.238+0000 7f125559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:04.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.238+0000 7f125cdab640 4 set_mon_vals no callback set 2024-09-13T11:59:04.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.238+0000 7f125559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:59:04.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.238+0000 7f125559a640 10 monclient: dump: 2024-09-13T11:59:04.252 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:04.253 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:04.253 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:04.253 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:04.253 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:04.253 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:04.253 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:04.253 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:04.253 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:04.253 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:04.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.239+0000 7f125cdab640 10 monclient: shutdown 2024-09-13T11:59:04.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.239+0000 7f125cdab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:04.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.239+0000 7f125cdab640 1 -- 172.21.15.100:0/2917699038 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12580a1ec0 msgr2=0x7f1258134ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:04.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.239+0000 7f125cdab640 1 --2- 172.21.15.100:0/2917699038 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12580a1ec0 0x7f1258134ae0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1250009e50 tx=0x7f125002a5f0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.254 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.239+0000 7f125cdab640 1 -- 172.21.15.100:0/2917699038 shutdown_connections 2024-09-13T11:59:04.254 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.239+0000 7f125cdab640 1 --2- 172.21.15.100:0/2917699038 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12580a1ec0 0x7f1258134ae0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.254 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.239+0000 7f125cdab640 1 --2- 172.21.15.100:0/2917699038 >> v2:172.21.15.122:3300/0 conn(0x7f12580a15a0 0x7f12580a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.254 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.239+0000 7f125cdab640 1 -- 172.21.15.100:0/2917699038 >> 172.21.15.100:0/2917699038 conn(0x7f12580a6470 msgr2=0x7f12580a2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:04.254 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.239+0000 7f125cdab640 1 -- 172.21.15.100:0/2917699038 shutdown_connections 2024-09-13T11:59:04.254 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.239+0000 7f125cdab640 1 -- 172.21.15.100:0/2917699038 wait complete. 2024-09-13T11:59:04.254 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.239+0000 7f125cdab640 10 monclient: build_initial_monmap 2024-09-13T11:59:04.254 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125cdab640 10 monclient: monmap: 2024-09-13T11:59:04.254 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:04.254 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:04.254 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:04.254 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:04.254 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:04.255 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:04.255 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:04.255 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:04.255 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:04.255 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:04.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125cdab640 1 Processor -- start 2024-09-13T11:59:04.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125cdab640 1 -- start start 2024-09-13T11:59:04.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125cdab640 10 monclient: init 2024-09-13T11:59:04.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125cdab640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:04.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125cdab640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:59:04.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125cdab640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f125809f6d0 0x7f12581c86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:04.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125cdab640 10 monclient(hunting): picked mon.noname-a con 0x7f125809f6d0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:04.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125cdab640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12580a15a0 0x7f12581c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:04.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125cdab640 10 monclient(hunting): picked mon.noname-c con 0x7f12580a15a0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:04.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125cdab640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12580a1ec0 0x7f125811ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:04.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125cdab640 10 monclient(hunting): picked mon.noname-b con 0x7f12580a1ec0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:04.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125cdab640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:04.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125cdab640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f12581351a0 con 0x7f125809f6d0 2024-09-13T11:59:04.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125cdab640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:04.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125cdab640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1258135020 con 0x7f12580a1ec0 2024-09-13T11:59:04.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125cdab640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:04.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125cdab640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f125809cfa0 con 0x7f12580a15a0 2024-09-13T11:59:04.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125cdab640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:04.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125cdab640 10 monclient(hunting): authenticate will time out at 1124.834849s 2024-09-13T11:59:04.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125659c640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f125809f6d0 0x7f12581c86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:04.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f1255d9b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12580a15a0 0x7f12581c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:04.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125659c640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f125809f6d0 0x7f12581c86c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47100/0 (socket says 172.21.15.100:47100) 2024-09-13T11:59:04.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f1255d9b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12580a15a0 0x7f12581c8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:51888/0 (socket says 172.21.15.100:51888) 2024-09-13T11:59:04.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125659c640 1 -- 172.21.15.100:0/2930176923 learned_addr learned my addr 172.21.15.100:0/2930176923 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:04.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125659c640 10 monclient(hunting): get_auth_request con 0x7f125809f6d0 auth_method 0 2024-09-13T11:59:04.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:04.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125659c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:04.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f125659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:04.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f1255d9b640 10 monclient(hunting): get_auth_request con 0x7f12580a15a0 auth_method 0 2024-09-13T11:59:04.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f1255d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:04.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f1255d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:04.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.240+0000 7f1255d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:04.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f125659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:04.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f125659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:04.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f125659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:04.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f1255d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:04.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f1255d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:04.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f1255d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:04.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f1256d9d640 1 --2- 172.21.15.100:0/2930176923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12580a1ec0 0x7f125811ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:04.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f1256d9d640 10 monclient(hunting): get_auth_request con 0x7f12580a1ec0 auth_method 0 2024-09-13T11:59:04.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f1256d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:04.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f1256d9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:04.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f1256d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:04.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f125659c640 10 monclient(hunting): handle_auth_done global_id 15501 payload 824 2024-09-13T11:59:04.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f125659c640 1 -- 172.21.15.100:0/2930176923 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12580a15a0 msgr2=0x7f12581c8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:04.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f125659c640 1 --2- 172.21.15.100:0/2930176923 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12580a15a0 0x7f12581c8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.259 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f125659c640 1 -- 172.21.15.100:0/2930176923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12580a1ec0 msgr2=0x7f125811ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:04.259 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f125659c640 1 --2- 172.21.15.100:0/2930176923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12580a1ec0 0x7f125811ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.259 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f125659c640 10 monclient: _finish_hunting 0 2024-09-13T11:59:04.259 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f125659c640 1 monclient: found mon.noname-a 2024-09-13T11:59:04.259 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f125659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:04.259 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f125659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:04.259 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f125659c640 1 -- 172.21.15.100:0/2930176923 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1258123a50 con 0x7f125809f6d0 2024-09-13T11:59:04.259 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f1255d9b640 1 --2- 172.21.15.100:0/2930176923 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12580a15a0 0x7f12581c8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:59:04.259 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f125659c640 1 --2- 172.21.15.100:0/2930176923 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f125809f6d0 0x7f12581c86c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f124800d800 tx=0x7f124800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:04.259 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f1246ffd640 1 -- 172.21.15.100:0/2930176923 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124800b6c0 con 0x7f125809f6d0 2024-09-13T11:59:04.259 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f1246ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:04.259 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f1246ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:04.259 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f1246ffd640 10 monclient: dump: 2024-09-13T11:59:04.260 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:04.260 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:04.260 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:04.260 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:04.260 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:04.260 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:04.260 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:04.260 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:04.260 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:04.260 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:04.260 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f1246ffd640 10 monclient: _finish_auth 0 2024-09-13T11:59:04.260 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f1246ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:59:04.260 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f1246ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:04.261 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f1246ffd640 1 -- 172.21.15.100:0/2930176923 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1248019070 con 0x7f125809f6d0 2024-09-13T11:59:04.261 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f125cdab640 5 monclient: authenticate success, global_id 15501 2024-09-13T11:59:04.261 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f1246ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:04.261 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f1246ffd640 1 -- 172.21.15.100:0/2930176923 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124800ba80 con 0x7f125809f6d0 2024-09-13T11:59:04.261 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f125cdab640 10 monclient: _renew_subs 2024-09-13T11:59:04.261 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f125cdab640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:04.261 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f125cdab640 1 -- 172.21.15.100:0/2930176923 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1258134020 con 0x7f125809f6d0 2024-09-13T11:59:04.261 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f1246ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:04.261 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f1246ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:04.261 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f1254d99640 4 set_mon_vals no callback set 2024-09-13T11:59:04.261 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f1246ffd640 10 monclient: dump: 2024-09-13T11:59:04.261 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:04.261 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:04.262 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:04.262 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:04.262 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:04.262 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:04.262 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:04.262 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:04.262 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:04.262 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:04.262 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f125cdab640 10 monclient: _renew_subs 2024-09-13T11:59:04.262 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f125cdab640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:04.262 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.241+0000 7f125cdab640 1 -- 172.21.15.100:0/2930176923 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1258122250 con 0x7f125809f6d0 2024-09-13T11:59:04.262 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.242+0000 7f1246ffd640 1 -- 172.21.15.100:0/2930176923 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f124800bc20 con 0x7f125809f6d0 2024-09-13T11:59:04.262 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.242+0000 7f125cdab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:04.263 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.242+0000 7f125cdab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:04.263 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.242+0000 7f125cdab640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:04.263 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.242+0000 7f125cdab640 1 -- 172.21.15.100:0/2930176923 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12580a1980 con 0x7f125809f6d0 2024-09-13T11:59:04.263 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.242+0000 7f1246ffd640 1 --2- 172.21.15.100:0/2930176923 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f123803dfd0 0x7f1238040490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:04.263 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.242+0000 7f1246ffd640 1 -- 172.21.15.100:0/2930176923 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7f1248052aa0 con 0x7f125809f6d0 2024-09-13T11:59:04.263 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.243+0000 7f1255d9b640 1 --2- 172.21.15.100:0/2930176923 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f123803dfd0 0x7f1238040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:04.263 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.243+0000 7f1255d9b640 10 monclient: get_auth_request con 0x7f123803dfd0 auth_method 0 2024-09-13T11:59:04.263 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.243+0000 7f1255d9b640 1 --2- 172.21.15.100:0/2930176923 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f123803dfd0 0x7f1238040490 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f12581230e0 tx=0x7f124c007cf0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:04.264 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.247+0000 7f1246ffd640 1 -- 172.21.15.100:0/2930176923 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f124801cd70 con 0x7f125809f6d0 2024-09-13T11:59:04.264 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.247+0000 7f1246ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:04.264 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.247+0000 7f1246ffd640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:59:04.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.486+0000 7f125cdab640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:59:04.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.486+0000 7f125cdab640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:59:04.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.486+0000 7f125cdab640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:04.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.486+0000 7f125cdab640 1 -- 172.21.15.100:0/2930176923 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f125806ca60 con 0x7f125809f6d0 2024-09-13T11:59:04.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.487+0000 7f1246ffd640 1 -- 172.21.15.100:0/2930176923 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f1248013390 con 0x7f125809f6d0 2024-09-13T11:59:04.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.487+0000 7f1246ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:59:04.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.487+0000 7f1246ffd640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:59:04.490 INFO:teuthology.orchestra.run.smithi100.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","modified":"2024-09-13T11:52:13.286120Z","created":"2024-09-13T11:52:13.286120Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:59:04.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.489+0000 7f125cdab640 1 -- 172.21.15.100:0/2930176923 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f123803dfd0 msgr2=0x7f1238040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:04.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.489+0000 7f125cdab640 1 --2- 172.21.15.100:0/2930176923 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f123803dfd0 0x7f1238040490 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f12581230e0 tx=0x7f124c007cf0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.489+0000 7f125cdab640 10 monclient: shutdown 2024-09-13T11:59:04.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.489+0000 7f125cdab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:04.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.489+0000 7f125cdab640 1 -- 172.21.15.100:0/2930176923 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f125809f6d0 msgr2=0x7f12581c86c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:04.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.489+0000 7f125cdab640 1 --2- 172.21.15.100:0/2930176923 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f125809f6d0 0x7f12581c86c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f124800d800 tx=0x7f124800dcc0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.489+0000 7f125cdab640 1 -- 172.21.15.100:0/2930176923 shutdown_connections 2024-09-13T11:59:04.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.489+0000 7f125cdab640 1 --2- 172.21.15.100:0/2930176923 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f123803dfd0 0x7f1238040490 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.489+0000 7f125cdab640 1 --2- 172.21.15.100:0/2930176923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12580a1ec0 0x7f125811ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.489+0000 7f125cdab640 1 --2- 172.21.15.100:0/2930176923 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f12580a15a0 0x7f12581c8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.489+0000 7f125cdab640 1 --2- 172.21.15.100:0/2930176923 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f125809f6d0 0x7f12581c86c0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.489+0000 7f125cdab640 1 -- 172.21.15.100:0/2930176923 >> 172.21.15.100:0/2930176923 conn(0x7f12580a6470 msgr2=0x7f12580a51b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:04.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.489+0000 7f125cdab640 1 -- 172.21.15.100:0/2930176923 shutdown_connections 2024-09-13T11:59:04.492 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.489+0000 7f125cdab640 1 -- 172.21.15.100:0/2930176923 wait complete. 2024-09-13T11:59:04.501 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-13T11:59:04.501 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-13T11:59:04.512 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T11:59:04.512 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:59:04.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.606+0000 7f4d1d544640 10 monclient: get_monmap_and_config 2024-09-13T11:59:04.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.606+0000 7f4d1d544640 10 monclient: build_initial_monmap 2024-09-13T11:59:04.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.606+0000 7f4d1d544640 10 monclient: monmap: 2024-09-13T11:59:04.609 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:04.609 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:04.609 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:04.608715+0000 2024-09-13T11:59:04.609 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:04.608715+0000 2024-09-13T11:59:04.609 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:04.609 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:04.609 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:04.610 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:04.610 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:04.610 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:04.610 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:04.610 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:04.618 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.607+0000 7f4d1d544640 1 Processor -- start 2024-09-13T11:59:04.618 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.607+0000 7f4d1d544640 1 -- start start 2024-09-13T11:59:04.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.607+0000 7f4d1d544640 10 monclient: init 2024-09-13T11:59:04.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.607+0000 7f4d1d544640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:04.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.607+0000 7f4d1d544640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-13T11:59:04.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.607+0000 7f4d1d544640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f4d18137a60 0x7f4d180a2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:04.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d1d544640 10 monclient(hunting): picked mon.noname-a con 0x7f4d18137a60 addr v2:172.21.15.100:3300/0 2024-09-13T11:59:04.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d1d544640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d18137e40 0x7f4d180a4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:04.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d1d544640 10 monclient(hunting): picked mon.noname-b con 0x7f4d18137e40 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:04.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d1d544640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d18138270 addr v1:172.21.15.122:6789/0 2024-09-13T11:59:04.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d1d544640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d180a5be0 con 0x7f4d18138270 2024-09-13T11:59:04.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d1d544640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:04.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d1d544640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f4d180a4410 con 0x7f4d18137a60 2024-09-13T11:59:04.619 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d1d544640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:04.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d1d544640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f4d180a76b0 con 0x7f4d18137e40 2024-09-13T11:59:04.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d1d544640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:04.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d1d544640 10 monclient(hunting): authenticate will time out at 1125.202067s 2024-09-13T11:59:04.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d0e7fc640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d18137e40 0x7f4d180a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:04.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d16ffd640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f4d18137a60 0x7f4d180a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:04.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d0e7fc640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d18137e40 0x7f4d180a4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:51894/0 (socket says 172.21.15.100:51894) 2024-09-13T11:59:04.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d16ffd640 10 monclient(hunting): get_auth_request con 0x7f4d18137a60 auth_method 0 2024-09-13T11:59:04.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d0e7fc640 1 -- 172.21.15.100:0/3872735759 learned_addr learned my addr 172.21.15.100:0/3872735759 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:04.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d16ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:04.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:04.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:04.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d0e7fc640 10 monclient(hunting): get_auth_request con 0x7f4d18137e40 auth_method 0 2024-09-13T11:59:04.620 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d0e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:04.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:04.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:04.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d16ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:04.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d16ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:04.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d16ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:04.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d0e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:04.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d0e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:04.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d0e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:04.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d16ffd640 10 monclient(hunting): handle_auth_done global_id 15504 payload 274 2024-09-13T11:59:04.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d16ffd640 1 -- 172.21.15.100:0/3872735759 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d18137e40 msgr2=0x7f4d180a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:04.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d16ffd640 1 --2- 172.21.15.100:0/3872735759 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d18137e40 0x7f4d180a4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d16ffd640 1 -- 172.21.15.100:0/3872735759 >> v1:172.21.15.122:6789/0 conn(0x7f4d18138270 legacy=0x7f4d180a4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:04.621 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d16ffd640 10 monclient: _finish_hunting 0 2024-09-13T11:59:04.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d16ffd640 1 monclient: found mon.noname-a 2024-09-13T11:59:04.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:04.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d16ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:04.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.608+0000 7f4d16ffd640 1 -- 172.21.15.100:0/3872735759 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d180a19e0 con 0x7f4d18137a60 2024-09-13T11:59:04.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d16ffd640 1 --2- 172.21.15.100:0/3872735759 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d18137a60 0x7f4d180a2980 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4d180a7880 tx=0x7f4d0802a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35412047b85c7de server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:04.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d167fc640 1 -- 172.21.15.100:0/3872735759 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d08037070 con 0x7f4d18137a60 2024-09-13T11:59:04.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:04.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:04.622 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d167fc640 10 monclient: dump: 2024-09-13T11:59:04.622 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:04.622 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:04.622 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:04.622 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:04.622 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:04.623 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:04.623 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:04.623 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:04.623 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:04.623 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:04.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d167fc640 10 monclient: _finish_auth 0 2024-09-13T11:59:04.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d167fc640 10 monclient: _check_auth_tickets 2024-09-13T11:59:04.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:04.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d1d544640 5 monclient: authenticate success, global_id 15504 2024-09-13T11:59:04.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d167fc640 1 -- 172.21.15.100:0/3872735759 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d0802adb0 con 0x7f4d18137a60 2024-09-13T11:59:04.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d1d544640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:04.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d167fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:04.623 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d1d544640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:04.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d1d544640 4 set_mon_vals no callback set 2024-09-13T11:59:04.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d167fc640 1 -- 172.21.15.100:0/3872735759 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d08032660 con 0x7f4d18137a60 2024-09-13T11:59:04.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:04.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:04.624 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d167fc640 10 monclient: dump: 2024-09-13T11:59:04.624 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:04.624 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:04.624 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:04.624 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:04.624 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:04.624 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:04.624 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:04.624 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:04.625 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:04.625 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:04.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d1d544640 10 monclient: shutdown 2024-09-13T11:59:04.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d1d544640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:04.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d1d544640 1 -- 172.21.15.100:0/3872735759 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d18137a60 msgr2=0x7f4d180a2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:04.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d1d544640 1 --2- 172.21.15.100:0/3872735759 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d18137a60 0x7f4d180a2980 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4d180a7880 tx=0x7f4d0802a060 comp rx=0 tx=0).stop 2024-09-13T11:59:04.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d1d544640 1 -- 172.21.15.100:0/3872735759 shutdown_connections 2024-09-13T11:59:04.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d1d544640 1 --2- 172.21.15.100:0/3872735759 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d18137e40 0x7f4d180a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d1d544640 1 --2- 172.21.15.100:0/3872735759 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d18137a60 0x7f4d180a2980 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d1d544640 1 -- 172.21.15.100:0/3872735759 >> 172.21.15.100:0/3872735759 conn(0x7f4d180aa580 msgr2=0x7f4d180a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:04.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.609+0000 7f4d1d544640 1 -- 172.21.15.100:0/3872735759 shutdown_connections 2024-09-13T11:59:04.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.610+0000 7f4d1d544640 1 -- 172.21.15.100:0/3872735759 wait complete. 2024-09-13T11:59:04.625 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.610+0000 7f4d1d544640 10 monclient: build_initial_monmap 2024-09-13T11:59:04.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.610+0000 7f4d1d544640 10 monclient: monmap: 2024-09-13T11:59:04.626 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:04.626 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:04.626 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:04.626 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:04.626 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:04.626 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:04.626 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:04.626 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:04.626 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:04.626 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:04.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.610+0000 7f4d1d544640 1 Processor -- start 2024-09-13T11:59:04.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d1d544640 1 -- start start 2024-09-13T11:59:04.626 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d1d544640 10 monclient: init 2024-09-13T11:59:04.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d1d544640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:04.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d1d544640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:59:04.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d1d544640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d18137a60 0x7f4d18133940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:04.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d1d544640 10 monclient(hunting): picked mon.noname-b con 0x7f4d18137a60 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:04.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d1d544640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d18137e40 0x7f4d18133e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:04.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d1d544640 10 monclient(hunting): picked mon.noname-a con 0x7f4d18137e40 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:04.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d1d544640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d18138270 0x7f4d18139610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:04.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d1d544640 10 monclient(hunting): picked mon.noname-c con 0x7f4d18138270 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:04.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d1d544640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:04.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d1d544640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f4d1806ff00 con 0x7f4d18137e40 2024-09-13T11:59:04.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d1d544640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:04.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d1d544640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f4d1806fd80 con 0x7f4d18137a60 2024-09-13T11:59:04.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d1d544640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:04.627 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d1d544640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f4d180a55a0 con 0x7f4d18138270 2024-09-13T11:59:04.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d1d544640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:04.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d1d544640 10 monclient(hunting): authenticate will time out at 1125.205463s 2024-09-13T11:59:04.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d0e7fc640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d18137e40 0x7f4d18133e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:04.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d0e7fc640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d18137e40 0x7f4d18133e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47104/0 (socket says 172.21.15.100:47104) 2024-09-13T11:59:04.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d0e7fc640 1 -- 172.21.15.100:0/1334321185 learned_addr learned my addr 172.21.15.100:0/1334321185 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:04.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d0e7fc640 10 monclient(hunting): get_auth_request con 0x7f4d18137e40 auth_method 0 2024-09-13T11:59:04.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d0e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:04.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:04.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:04.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d16ffd640 1 --2- 172.21.15.100:0/1334321185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d18137a60 0x7f4d18133940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:04.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d177fe640 1 --2- 172.21.15.100:0/1334321185 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d18138270 0x7f4d18139610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:04.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d16ffd640 10 monclient(hunting): get_auth_request con 0x7f4d18137a60 auth_method 0 2024-09-13T11:59:04.628 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d16ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:04.629 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:04.629 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:04.629 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d177fe640 10 monclient(hunting): get_auth_request con 0x7f4d18138270 auth_method 0 2024-09-13T11:59:04.629 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:04.629 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d177fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:04.629 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:04.629 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d0e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:04.629 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d0e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:04.629 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d0e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:04.629 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:04.629 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:04.629 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:04.629 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d16ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:04.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d16ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:04.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d16ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:04.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d0e7fc640 10 monclient(hunting): handle_auth_done global_id 15507 payload 824 2024-09-13T11:59:04.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d0e7fc640 1 -- 172.21.15.100:0/1334321185 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d18138270 msgr2=0x7f4d18139610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:04.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d0e7fc640 1 --2- 172.21.15.100:0/1334321185 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d18138270 0x7f4d18139610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d0e7fc640 1 -- 172.21.15.100:0/1334321185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d18137a60 msgr2=0x7f4d18133940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:04.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d0e7fc640 1 --2- 172.21.15.100:0/1334321185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d18137a60 0x7f4d18133940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d0e7fc640 10 monclient: _finish_hunting 0 2024-09-13T11:59:04.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d0e7fc640 1 monclient: found mon.noname-a 2024-09-13T11:59:04.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:04.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d0e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:04.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.611+0000 7f4d0e7fc640 1 -- 172.21.15.100:0/1334321185 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d1813cb00 con 0x7f4d18137e40 2024-09-13T11:59:04.630 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.612+0000 7f4d0e7fc640 1 --2- 172.21.15.100:0/1334321185 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d18137e40 0x7f4d18133e80 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4d0000e8b0 tx=0x7f4d0000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:04.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.612+0000 7f4d0ffff640 1 -- 172.21.15.100:0/1334321185 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d00017070 con 0x7f4d18137e40 2024-09-13T11:59:04.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.612+0000 7f4d0ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:04.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.612+0000 7f4d0ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:04.631 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.612+0000 7f4d0ffff640 10 monclient: dump: 2024-09-13T11:59:04.631 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:04.631 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:04.631 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:04.631 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:04.631 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:04.631 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:04.631 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:04.631 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:04.631 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:04.631 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:04.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.612+0000 7f4d0ffff640 10 monclient: _finish_auth 0 2024-09-13T11:59:04.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.612+0000 7f4d0ffff640 10 monclient: _check_auth_tickets 2024-09-13T11:59:04.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.612+0000 7f4d0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:04.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.612+0000 7f4d0ffff640 1 -- 172.21.15.100:0/1334321185 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d0000c8e0 con 0x7f4d18137e40 2024-09-13T11:59:04.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.612+0000 7f4d1d544640 5 monclient: authenticate success, global_id 15507 2024-09-13T11:59:04.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.612+0000 7f4d1d544640 10 monclient: _renew_subs 2024-09-13T11:59:04.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.612+0000 7f4d1d544640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:04.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.612+0000 7f4d1d544640 1 -- 172.21.15.100:0/1334321185 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d180a1140 con 0x7f4d18137e40 2024-09-13T11:59:04.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.612+0000 7f4d0ffff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:04.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.612+0000 7f4d0ffff640 1 -- 172.21.15.100:0/1334321185 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0000ccb0 con 0x7f4d18137e40 2024-09-13T11:59:04.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.612+0000 7f4d0ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:04.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.612+0000 7f4d0ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:04.632 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.612+0000 7f4d0ffff640 10 monclient: dump: 2024-09-13T11:59:04.633 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:04.633 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:04.633 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:04.633 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:04.633 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:04.633 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:04.633 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:04.633 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:04.633 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:04.633 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:04.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.612+0000 7f4d1d544640 10 monclient: _renew_subs 2024-09-13T11:59:04.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.612+0000 7f4d15ffb640 4 set_mon_vals no callback set 2024-09-13T11:59:04.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.612+0000 7f4d1d544640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:04.633 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.612+0000 7f4d1d544640 1 -- 172.21.15.100:0/1334321185 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d1813d3a0 con 0x7f4d18137e40 2024-09-13T11:59:04.634 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.613+0000 7f4d1d544640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:04.634 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.613+0000 7f4d1d544640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:04.634 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.613+0000 7f4d1d544640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:04.634 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.613+0000 7f4d1d544640 1 -- 172.21.15.100:0/1334321185 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d181344e0 con 0x7f4d18137e40 2024-09-13T11:59:04.634 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.613+0000 7f4d0ffff640 1 -- 172.21.15.100:0/1334321185 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f4d0001a070 con 0x7f4d18137e40 2024-09-13T11:59:04.634 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.613+0000 7f4d0ffff640 1 --2- 172.21.15.100:0/1334321185 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f4cf803dfd0 0x7f4cf8040490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:04.634 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.613+0000 7f4d0ffff640 1 -- 172.21.15.100:0/1334321185 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7f4d000524f0 con 0x7f4d18137e40 2024-09-13T11:59:04.634 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.613+0000 7f4d16ffd640 1 --2- 172.21.15.100:0/1334321185 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f4cf803dfd0 0x7f4cf8040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:04.634 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.613+0000 7f4d16ffd640 10 monclient: get_auth_request con 0x7f4cf803dfd0 auth_method 0 2024-09-13T11:59:04.636 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.613+0000 7f4d16ffd640 1 --2- 172.21.15.100:0/1334321185 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f4cf803dfd0 0x7f4cf8040490 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4d08034000 tx=0x7f4d0802e000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:04.637 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.617+0000 7f4d0ffff640 1 -- 172.21.15.100:0/1334321185 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f4d1813cb00 con 0x7f4d18137e40 2024-09-13T11:59:04.637 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.617+0000 7f4d0ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:04.637 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.617+0000 7f4d0ffff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:59:04.789 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.786+0000 7f4d1d544640 1 -- 172.21.15.100:0/1334321185 --> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4d180a2980 con 0x7f4cf803dfd0 2024-09-13T11:59:04.790 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.788+0000 7f4d0ffff640 1 -- 172.21.15.100:0/1334321185 <== mgr.4114 v2:172.21.15.122:6824/441789830 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34013 (secure 0 0 0) 0x7f4d180a2980 con 0x7f4cf803dfd0 2024-09-13T11:59:04.790 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:59:04.791 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-13T11:59:04.792 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.790+0000 7f4d1d544640 1 -- 172.21.15.100:0/1334321185 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f4cf803dfd0 msgr2=0x7f4cf8040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:04.792 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.790+0000 7f4d1d544640 1 --2- 172.21.15.100:0/1334321185 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f4cf803dfd0 0x7f4cf8040490 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4d08034000 tx=0x7f4d0802e000 comp rx=0 tx=0).stop 2024-09-13T11:59:04.793 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.790+0000 7f4d1d544640 10 monclient: shutdown 2024-09-13T11:59:04.793 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.790+0000 7f4d1d544640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:04.793 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.790+0000 7f4d1d544640 1 -- 172.21.15.100:0/1334321185 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d18137e40 msgr2=0x7f4d18133e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:04.793 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.790+0000 7f4d1d544640 1 --2- 172.21.15.100:0/1334321185 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d18137e40 0x7f4d18133e80 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4d0000e8b0 tx=0x7f4d0000ed70 comp rx=0 tx=0).stop 2024-09-13T11:59:04.793 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.790+0000 7f4d1d544640 1 -- 172.21.15.100:0/1334321185 shutdown_connections 2024-09-13T11:59:04.793 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.790+0000 7f4d1d544640 1 --2- 172.21.15.100:0/1334321185 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f4cf803dfd0 0x7f4cf8040490 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.793 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.790+0000 7f4d1d544640 1 --2- 172.21.15.100:0/1334321185 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d18137e40 0x7f4d18133e80 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.793 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.790+0000 7f4d1d544640 1 --2- 172.21.15.100:0/1334321185 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d18138270 0x7f4d18139610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.793 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.790+0000 7f4d1d544640 1 --2- 172.21.15.100:0/1334321185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d18137a60 0x7f4d18133940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.794 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.790+0000 7f4d1d544640 1 -- 172.21.15.100:0/1334321185 >> 172.21.15.100:0/1334321185 conn(0x7f4d180aa580 msgr2=0x7f4d180a8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:04.794 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.790+0000 7f4d1d544640 1 -- 172.21.15.100:0/1334321185 shutdown_connections 2024-09-13T11:59:04.794 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.790+0000 7f4d1d544640 1 -- 172.21.15.100:0/1334321185 wait complete. 2024-09-13T11:59:04.802 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-13T11:59:03.285569+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":211864,"kb_used_data":24728,"kb_used_omap":9,"kb_used_meta":186998,"kb_avail":566019176,"statfs":{"total":579820584960,"available":579603636224,"internally_reserved":0,"allocated":25321472,"data_stored":10410820,"data_compressed":59246,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":191486624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002229"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":599,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.972852+0000","last_change":"2024-09-13T11:53:06.266756+0000","last_active":"2024-09-13T11:58:48.972852+0000","last_peered":"2024-09-13T11:58:48.972852+0000","last_clean":"2024-09-13T11:58:48.972852+0000","last_became_active":"2024-09-13T11:53:04.566167+0000","last_became_peered":"2024-09-13T11:53:04.566167+0000","last_unstale":"2024-09-13T11:58:48.972852+0000","last_undegraded":"2024-09-13T11:58:48.972852+0000","last_fullsized":"2024-09-13T11:58:48.972852+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:41.983816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084363899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":589,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.070903+0000","last_change":"2024-09-13T11:53:06.266346+0000","last_active":"2024-09-13T11:58:48.070903+0000","last_peered":"2024-09-13T11:58:48.070903+0000","last_clean":"2024-09-13T11:58:48.070903+0000","last_became_active":"2024-09-13T11:53:04.258975+0000","last_became_peered":"2024-09-13T11:53:04.258975+0000","last_unstale":"2024-09-13T11:58:48.070903+0000","last_undegraded":"2024-09-13T11:58:48.070903+0000","last_fullsized":"2024-09-13T11:58:48.070903+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:36.773393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055464299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":599,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.972511+0000","last_change":"2024-09-13T11:53:06.266446+0000","last_active":"2024-09-13T11:58:48.972511+0000","last_peered":"2024-09-13T11:58:48.972511+0000","last_clean":"2024-09-13T11:58:48.972511+0000","last_became_active":"2024-09-13T11:53:04.262475+0000","last_became_peered":"2024-09-13T11:53:04.262475+0000","last_unstale":"2024-09-13T11:58:48.972511+0000","last_undegraded":"2024-09-13T11:58:48.972511+0000","last_fullsized":"2024-09-13T11:58:48.972511+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:24:12.639511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055213099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":589,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.068856+0000","last_change":"2024-09-13T11:53:06.266390+0000","last_active":"2024-09-13T11:58:48.068856+0000","last_peered":"2024-09-13T11:58:48.068856+0000","last_clean":"2024-09-13T11:58:48.068856+0000","last_became_active":"2024-09-13T11:53:04.262392+0000","last_became_peered":"2024-09-13T11:53:04.262392+0000","last_unstale":"2024-09-13T11:58:48.068856+0000","last_undegraded":"2024-09-13T11:58:48.068856+0000","last_fullsized":"2024-09-13T11:58:48.068856+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:45.666214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057483700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":587,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.896615+0000","last_change":"2024-09-13T11:53:06.272635+0000","last_active":"2024-09-13T11:58:48.896615+0000","last_peered":"2024-09-13T11:58:48.896615+0000","last_clean":"2024-09-13T11:58:48.896615+0000","last_became_active":"2024-09-13T11:53:04.264659+0000","last_became_peered":"2024-09-13T11:53:04.264659+0000","last_unstale":"2024-09-13T11:58:48.896615+0000","last_undegraded":"2024-09-13T11:58:48.896615+0000","last_fullsized":"2024-09-13T11:58:48.896615+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:53:49.774126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010464509999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":596,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.670015+0000","last_change":"2024-09-13T11:53:06.266298+0000","last_active":"2024-09-13T11:58:48.670015+0000","last_peered":"2024-09-13T11:58:48.670015+0000","last_clean":"2024-09-13T11:58:48.670015+0000","last_became_active":"2024-09-13T11:53:04.270146+0000","last_became_peered":"2024-09-13T11:53:04.270146+0000","last_unstale":"2024-09-13T11:58:48.670015+0000","last_undegraded":"2024-09-13T11:58:48.670015+0000","last_fullsized":"2024-09-13T11:58:48.670015+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:38.439658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061806000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":599,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.972501+0000","last_change":"2024-09-13T11:53:06.266345+0000","last_active":"2024-09-13T11:58:48.972501+0000","last_peered":"2024-09-13T11:58:48.972501+0000","last_clean":"2024-09-13T11:58:48.972501+0000","last_became_active":"2024-09-13T11:53:04.262197+0000","last_became_peered":"2024-09-13T11:53:04.262197+0000","last_unstale":"2024-09-13T11:58:48.972501+0000","last_undegraded":"2024-09-13T11:58:48.972501+0000","last_fullsized":"2024-09-13T11:58:48.972501+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:05:51.189944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044149899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":586,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.896726+0000","last_change":"2024-09-13T11:53:06.272071+0000","last_active":"2024-09-13T11:58:48.896726+0000","last_peered":"2024-09-13T11:58:48.896726+0000","last_clean":"2024-09-13T11:58:48.896726+0000","last_became_active":"2024-09-13T11:53:04.270138+0000","last_became_peered":"2024-09-13T11:53:04.270138+0000","last_unstale":"2024-09-13T11:58:48.896726+0000","last_undegraded":"2024-09-13T11:58:48.896726+0000","last_fullsized":"2024-09-13T11:58:48.896726+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:17.090753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047405,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":691,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.973102+0000","last_change":"2024-09-13T11:52:57.706946+0000","last_active":"2024-09-13T11:58:48.973102+0000","last_peered":"2024-09-13T11:58:48.973102+0000","last_clean":"2024-09-13T11:58:48.973102+0000","last_became_active":"2024-09-13T11:52:57.705894+0000","last_became_peered":"2024-09-13T11:52:57.705894+0000","last_unstale":"2024-09-13T11:58:48.973102+0000","last_undegraded":"2024-09-13T11:58:48.973102+0000","last_fullsized":"2024-09-13T11:58:48.973102+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:52:56.225375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:52:56.225375+0000","last_clean_scrub_stamp":"2024-09-13T11:52:56.225375+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:05.299261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59246,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607629,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31128,"kb_used_data":4100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340712,"statfs":{"total":96636764160,"available":96604889088,"internally_reserved":0,"allocated":4198400,"data_stored":1558821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607628,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35348,"kb_used_data":4160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336492,"statfs":{"total":96636764160,"available":96600567808,"internally_reserved":0,"allocated":4259840,"data_stored":2087749,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607630,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35348,"kb_used_data":4160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336492,"statfs":{"total":96636764160,"available":96600567808,"internally_reserved":0,"allocated":4259840,"data_stored":2087749,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640333,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31136,"kb_used_data":4104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340704,"statfs":{"total":96636764160,"available":96604880896,"internally_reserved":0,"allocated":4202496,"data_stored":1558840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640332,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31132,"kb_used_data":4100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340708,"statfs":{"total":96636764160,"available":96604884992,"internally_reserved":0,"allocated":4198400,"data_stored":1558821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640333,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47772,"kb_used_data":4104,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94324068,"statfs":{"total":96636764160,"available":96587845632,"internally_reserved":0,"allocated":4202496,"data_stored":1558840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:59:04.803 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:59:04.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.897+0000 7ff3656b0640 10 monclient: get_monmap_and_config 2024-09-13T11:59:04.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.897+0000 7ff3656b0640 10 monclient: build_initial_monmap 2024-09-13T11:59:04.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.897+0000 7ff3656b0640 10 monclient: monmap: 2024-09-13T11:59:04.900 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:04.900 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:04.900 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:04.899626+0000 2024-09-13T11:59:04.900 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:04.899626+0000 2024-09-13T11:59:04.900 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:04.900 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:04.900 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:04.900 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:04.900 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:04.901 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:04.901 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:04.901 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:04.907 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.898+0000 7ff3656b0640 1 Processor -- start 2024-09-13T11:59:04.907 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.898+0000 7ff3656b0640 1 -- start start 2024-09-13T11:59:04.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.898+0000 7ff3656b0640 10 monclient: init 2024-09-13T11:59:04.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.898+0000 7ff3656b0640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:04.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.898+0000 7ff3656b0640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-13T11:59:04.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.898+0000 7ff3656b0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff360137a60 addr v1:172.21.15.100:6789/0 2024-09-13T11:59:04.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.898+0000 7ff3656b0640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff3600a2980 0x7ff36013c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:04.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.898+0000 7ff3656b0640 10 monclient(hunting): picked mon.noname-a con 0x7ff3600a2980 addr v2:172.21.15.100:3300/0 2024-09-13T11:59:04.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.898+0000 7ff3656b0640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff36013ce90 0x7ff36013f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:04.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.898+0000 7ff3656b0640 10 monclient(hunting): picked mon.noname-b con 0x7ff36013ce90 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:04.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.898+0000 7ff3656b0640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff360138550 con 0x7ff360137a60 2024-09-13T11:59:04.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.898+0000 7ff3656b0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:04.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.898+0000 7ff3656b0640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7ff3600a4410 con 0x7ff3600a2980 2024-09-13T11:59:04.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.898+0000 7ff3656b0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:04.909 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.898+0000 7ff3656b0640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7ff3600a5be0 con 0x7ff36013ce90 2024-09-13T11:59:04.909 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.898+0000 7ff3656b0640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:04.909 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.898+0000 7ff3656b0640 10 monclient(hunting): authenticate will time out at 1125.492994s 2024-09-13T11:59:04.909 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff357fff640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff3600a2980 0x7ff36013c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:04.909 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff35f7fe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff36013ce90 0x7ff36013f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:04.909 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff35effd640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7ff360137a60 0x7ff360137e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:57076/0 (socket says 172.21.15.100:57076) 2024-09-13T11:59:04.909 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff35effd640 1 -- 172.21.15.100:0/3345203128 learned_addr learned my addr 172.21.15.100:0/3345203128 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:59:04.909 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff357fff640 10 monclient(hunting): get_auth_request con 0x7ff3600a2980 auth_method 0 2024-09-13T11:59:04.909 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff357fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:04.909 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff357fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:04.909 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff357fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:04.909 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff35f7fe640 10 monclient(hunting): get_auth_request con 0x7ff36013ce90 auth_method 0 2024-09-13T11:59:04.909 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff35f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:04.909 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff35f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:04.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff35f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:04.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff357fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:04.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff357fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:04.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff357fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:04.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff35f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:04.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff35f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:04.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff35f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:04.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff357fff640 10 monclient(hunting): handle_auth_done global_id 15510 payload 274 2024-09-13T11:59:04.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff357fff640 1 -- 172.21.15.100:0/3345203128 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff36013ce90 msgr2=0x7ff36013f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:04.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff357fff640 1 --2- 172.21.15.100:0/3345203128 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff36013ce90 0x7ff36013f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff357fff640 1 -- 172.21.15.100:0/3345203128 >> v1:172.21.15.100:6789/0 conn(0x7ff360137a60 legacy=0x7ff360137e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:04.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff357fff640 10 monclient: _finish_hunting 0 2024-09-13T11:59:04.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff357fff640 1 monclient: found mon.noname-a 2024-09-13T11:59:04.911 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff357fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:04.911 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff357fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:04.911 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff357fff640 1 -- 172.21.15.100:0/3345203128 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3600a76b0 con 0x7ff3600a2980 2024-09-13T11:59:04.911 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff35e7fc640 1 -- 172.21.15.100:0/3345203128 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2476317707 0 0) 0x7ff360138550 con 0x7ff360137a60 2024-09-13T11:59:04.911 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff35e7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:59:04.911 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.899+0000 7ff35f7fe640 1 --2- 172.21.15.100:0/3345203128 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff36013ce90 0x7ff36013f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:59:04.911 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff357fff640 1 --2- 172.21.15.100:0/3345203128 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3600a2980 0x7ff36013c950 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff3480097a0 tx=0x7ff34802a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdd2d9b3c45cd9ef server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:04.911 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff35e7fc640 1 -- 172.21.15.100:0/3345203128 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff348037070 con 0x7ff3600a2980 2024-09-13T11:59:04.911 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff35e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:04.911 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff35e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:04.911 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff35e7fc640 10 monclient: dump: 2024-09-13T11:59:04.911 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:04.911 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:04.912 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:04.912 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:04.912 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:04.912 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:04.912 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:04.912 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:04.912 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:04.912 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:04.912 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff35e7fc640 10 monclient: _finish_auth 0 2024-09-13T11:59:04.912 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff35e7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:59:04.912 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff35e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:04.912 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff35e7fc640 1 -- 172.21.15.100:0/3345203128 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff34802adb0 con 0x7ff3600a2980 2024-09-13T11:59:04.912 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff3656b0640 5 monclient: authenticate success, global_id 15510 2024-09-13T11:59:04.912 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff3656b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:04.913 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff35e7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:04.913 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff3656b0640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:04.913 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff35e7fc640 1 -- 172.21.15.100:0/3345203128 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff348032660 con 0x7ff3600a2980 2024-09-13T11:59:04.913 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff3656b0640 4 set_mon_vals no callback set 2024-09-13T11:59:04.913 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff35e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:04.913 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff35e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:04.913 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff35e7fc640 10 monclient: dump: 2024-09-13T11:59:04.913 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:04.913 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:04.913 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:04.913 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:04.913 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:04.913 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:04.914 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:04.914 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:04.914 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:04.914 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:04.914 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff3656b0640 10 monclient: shutdown 2024-09-13T11:59:04.914 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff3656b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:04.914 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff3656b0640 1 -- 172.21.15.100:0/3345203128 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3600a2980 msgr2=0x7ff36013c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:04.914 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff3656b0640 1 --2- 172.21.15.100:0/3345203128 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3600a2980 0x7ff36013c950 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff3480097a0 tx=0x7ff34802a060 comp rx=0 tx=0).stop 2024-09-13T11:59:04.914 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff3656b0640 1 -- 172.21.15.100:0/3345203128 shutdown_connections 2024-09-13T11:59:04.914 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff3656b0640 1 --2- 172.21.15.100:0/3345203128 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3600a2980 0x7ff36013c950 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.914 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff3656b0640 1 --2- 172.21.15.100:0/3345203128 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff36013ce90 0x7ff36013f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.914 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff3656b0640 1 -- 172.21.15.100:0/3345203128 >> 172.21.15.100:0/3345203128 conn(0x7ff3600aa580 msgr2=0x7ff3600a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:04.914 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff3656b0640 1 -- 172.21.15.100:0/3345203128 shutdown_connections 2024-09-13T11:59:04.915 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.900+0000 7ff3656b0640 1 -- 172.21.15.100:0/3345203128 wait complete. 2024-09-13T11:59:04.915 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.901+0000 7ff3656b0640 10 monclient: build_initial_monmap 2024-09-13T11:59:04.915 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.901+0000 7ff3656b0640 10 monclient: monmap: 2024-09-13T11:59:04.915 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:04.915 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:04.915 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:04.915 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:04.915 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:04.915 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:04.915 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:04.915 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:04.915 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:04.915 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:04.915 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.901+0000 7ff3656b0640 1 Processor -- start 2024-09-13T11:59:04.916 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.901+0000 7ff3656b0640 1 -- start start 2024-09-13T11:59:04.916 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.901+0000 7ff3656b0640 10 monclient: init 2024-09-13T11:59:04.916 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3656b0640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:04.916 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3656b0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:59:04.916 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3656b0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3600a2980 0x7ff3601361b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:04.916 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3656b0640 10 monclient(hunting): picked mon.noname-b con 0x7ff3600a2980 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:04.916 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3656b0640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff360137a60 0x7ff3601366f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:04.916 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3656b0640 10 monclient(hunting): picked mon.noname-c con 0x7ff360137a60 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:04.916 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3656b0640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff36013ce90 0x7ff36009d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:04.916 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3656b0640 10 monclient(hunting): picked mon.noname-a con 0x7ff36013ce90 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:04.916 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3656b0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:04.916 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3656b0640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7ff36006fd80 con 0x7ff36013ce90 2024-09-13T11:59:04.916 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3656b0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:04.917 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3656b0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff3600a1700 con 0x7ff3600a2980 2024-09-13T11:59:04.917 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3656b0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:04.917 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3656b0640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7ff36006ff00 con 0x7ff360137a60 2024-09-13T11:59:04.917 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3656b0640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:04.917 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3656b0640 10 monclient(hunting): authenticate will time out at 1125.496319s 2024-09-13T11:59:04.917 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff357fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff360137a60 0x7ff3601366f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:04.917 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff35f7fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff36013ce90 0x7ff36009d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:04.917 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff357fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff360137a60 0x7ff3601366f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:51934/0 (socket says 172.21.15.100:51934) 2024-09-13T11:59:04.917 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff357fff640 1 -- 172.21.15.100:0/1732248462 learned_addr learned my addr 172.21.15.100:0/1732248462 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:04.917 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff35f7fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff36013ce90 0x7ff36009d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47132/0 (socket says 172.21.15.100:47132) 2024-09-13T11:59:04.917 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff357fff640 10 monclient(hunting): get_auth_request con 0x7ff360137a60 auth_method 0 2024-09-13T11:59:04.917 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff357fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:04.917 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff357fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:04.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff357fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:04.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff35f7fe640 10 monclient(hunting): get_auth_request con 0x7ff36013ce90 auth_method 0 2024-09-13T11:59:04.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff35f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:04.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff35f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:04.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff35f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:04.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff357fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:04.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff357fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:04.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff357fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:04.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff35f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:04.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff35f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:04.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff35f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:04.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff357fff640 10 monclient(hunting): handle_auth_done global_id 24608 payload 824 2024-09-13T11:59:04.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff357fff640 1 -- 172.21.15.100:0/1732248462 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3600a2980 msgr2=0x7ff3601361b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:04.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff357fff640 1 --2- 172.21.15.100:0/1732248462 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3600a2980 0x7ff3601361b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff357fff640 1 -- 172.21.15.100:0/1732248462 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff36013ce90 msgr2=0x7ff36009d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:04.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff357fff640 1 --2- 172.21.15.100:0/1732248462 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff36013ce90 0x7ff36009d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:04.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff357fff640 10 monclient: _finish_hunting 0 2024-09-13T11:59:04.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff357fff640 1 monclient: found mon.noname-c 2024-09-13T11:59:04.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff357fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:04.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff357fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:59:04.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff357fff640 1 -- 172.21.15.100:0/1732248462 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff360120b40 con 0x7ff360137a60 2024-09-13T11:59:04.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff357fff640 1 --2- 172.21.15.100:0/1732248462 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff360137a60 0x7ff3601366f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff348000c00 tx=0x7ff3480046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:04.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3577fe640 1 -- 172.21.15.100:0/1732248462 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff348041070 con 0x7ff360137a60 2024-09-13T11:59:04.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:04.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:59:04.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3577fe640 10 monclient: dump: 2024-09-13T11:59:04.919 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:04.920 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:04.920 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:04.920 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:04.920 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:04.920 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:04.920 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:04.920 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:04.920 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:04.920 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:04.920 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3577fe640 10 monclient: _finish_auth 0 2024-09-13T11:59:04.920 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3577fe640 10 monclient: _check_auth_tickets 2024-09-13T11:59:04.920 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:04.920 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3577fe640 1 -- 172.21.15.100:0/1732248462 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff34803c460 con 0x7ff360137a60 2024-09-13T11:59:04.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.902+0000 7ff3656b0640 5 monclient: authenticate success, global_id 24608 2024-09-13T11:59:04.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.903+0000 7ff3656b0640 10 monclient: _renew_subs 2024-09-13T11:59:04.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.903+0000 7ff3656b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:59:04.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.903+0000 7ff3656b0640 1 -- 172.21.15.100:0/1732248462 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff36013eac0 con 0x7ff360137a60 2024-09-13T11:59:04.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.903+0000 7ff3577fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:04.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.903+0000 7ff3577fe640 1 -- 172.21.15.100:0/1732248462 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff348037040 con 0x7ff360137a60 2024-09-13T11:59:04.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.903+0000 7ff3577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:04.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.903+0000 7ff3577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:59:04.921 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.903+0000 7ff3577fe640 10 monclient: dump: 2024-09-13T11:59:04.921 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:04.921 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:04.921 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:04.921 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:04.922 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:04.922 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:04.922 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:04.922 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:04.922 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:04.922 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:04.922 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.903+0000 7ff35dffb640 4 set_mon_vals no callback set 2024-09-13T11:59:04.922 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.903+0000 7ff3656b0640 10 monclient: _renew_subs 2024-09-13T11:59:04.922 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.903+0000 7ff3656b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:59:04.922 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.903+0000 7ff3656b0640 1 -- 172.21.15.100:0/1732248462 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff360121100 con 0x7ff360137a60 2024-09-13T11:59:04.922 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.904+0000 7ff3577fe640 1 -- 172.21.15.100:0/1732248462 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7ff348044070 con 0x7ff360137a60 2024-09-13T11:59:04.922 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.904+0000 7ff3656b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:04.922 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.904+0000 7ff3656b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:04.922 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.904+0000 7ff3656b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:59:04.923 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.904+0000 7ff3656b0640 1 -- 172.21.15.100:0/1732248462 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff360137e40 con 0x7ff360137a60 2024-09-13T11:59:04.923 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.904+0000 7ff3577fe640 1 --2- 172.21.15.100:0/1732248462 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff34003dfd0 0x7ff340040490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:04.923 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.904+0000 7ff3577fe640 1 -- 172.21.15.100:0/1732248462 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7ff348072e10 con 0x7ff360137a60 2024-09-13T11:59:04.923 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.904+0000 7ff35effd640 1 --2- 172.21.15.100:0/1732248462 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff34003dfd0 0x7ff340040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:04.923 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.904+0000 7ff35effd640 10 monclient: get_auth_request con 0x7ff34003dfd0 auth_method 0 2024-09-13T11:59:04.923 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.905+0000 7ff35effd640 1 --2- 172.21.15.100:0/1732248462 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff34003dfd0 0x7ff340040490 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff3500015d0 tx=0x7ff3500078d0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:04.923 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.908+0000 7ff3577fe640 1 -- 172.21.15.100:0/1732248462 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7ff34803ac60 con 0x7ff360137a60 2024-09-13T11:59:04.924 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.908+0000 7ff3577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:04.924 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:04.908+0000 7ff3577fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:59:05.083 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.081+0000 7ff3656b0640 1 -- 172.21.15.100:0/1732248462 --> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff36006a0f0 con 0x7ff34003dfd0 2024-09-13T11:59:05.084 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.083+0000 7ff3577fe640 1 -- 172.21.15.100:0/1732248462 <== mgr.4114 v2:172.21.15.122:6824/441789830 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34013 (secure 0 0 0) 0x7ff36006a0f0 con 0x7ff34003dfd0 2024-09-13T11:59:05.085 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:59:05.085 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-13T11:59:05.087 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.085+0000 7ff3656b0640 1 -- 172.21.15.100:0/1732248462 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff34003dfd0 msgr2=0x7ff340040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:05.087 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.085+0000 7ff3656b0640 1 --2- 172.21.15.100:0/1732248462 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff34003dfd0 0x7ff340040490 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff3500015d0 tx=0x7ff3500078d0 comp rx=0 tx=0).stop 2024-09-13T11:59:05.088 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.085+0000 7ff3656b0640 10 monclient: shutdown 2024-09-13T11:59:05.088 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.085+0000 7ff3656b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:05.088 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.085+0000 7ff3656b0640 1 -- 172.21.15.100:0/1732248462 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff360137a60 msgr2=0x7ff3601366f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:05.088 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.085+0000 7ff3656b0640 1 --2- 172.21.15.100:0/1732248462 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff360137a60 0x7ff3601366f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff348000c00 tx=0x7ff3480046c0 comp rx=0 tx=0).stop 2024-09-13T11:59:05.088 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.085+0000 7ff3656b0640 1 -- 172.21.15.100:0/1732248462 shutdown_connections 2024-09-13T11:59:05.088 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.085+0000 7ff3656b0640 1 --2- 172.21.15.100:0/1732248462 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff34003dfd0 0x7ff340040490 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:05.088 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.085+0000 7ff3656b0640 1 --2- 172.21.15.100:0/1732248462 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff36013ce90 0x7ff36009d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:05.088 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.085+0000 7ff3656b0640 1 --2- 172.21.15.100:0/1732248462 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff360137a60 0x7ff3601366f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:05.088 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.085+0000 7ff3656b0640 1 --2- 172.21.15.100:0/1732248462 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3600a2980 0x7ff3601361b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:05.088 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.085+0000 7ff3656b0640 1 -- 172.21.15.100:0/1732248462 >> 172.21.15.100:0/1732248462 conn(0x7ff3600aa580 msgr2=0x7ff36013ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:05.088 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.085+0000 7ff3656b0640 1 -- 172.21.15.100:0/1732248462 shutdown_connections 2024-09-13T11:59:05.088 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.085+0000 7ff3656b0640 1 -- 172.21.15.100:0/1732248462 wait complete. 2024-09-13T11:59:05.097 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-13T11:59:03.285569+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":211864,"kb_used_data":24728,"kb_used_omap":9,"kb_used_meta":186998,"kb_avail":566019176,"statfs":{"total":579820584960,"available":579603636224,"internally_reserved":0,"allocated":25321472,"data_stored":10410820,"data_compressed":59246,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":191486624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002229"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":599,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.972852+0000","last_change":"2024-09-13T11:53:06.266756+0000","last_active":"2024-09-13T11:58:48.972852+0000","last_peered":"2024-09-13T11:58:48.972852+0000","last_clean":"2024-09-13T11:58:48.972852+0000","last_became_active":"2024-09-13T11:53:04.566167+0000","last_became_peered":"2024-09-13T11:53:04.566167+0000","last_unstale":"2024-09-13T11:58:48.972852+0000","last_undegraded":"2024-09-13T11:58:48.972852+0000","last_fullsized":"2024-09-13T11:58:48.972852+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:41.983816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084363899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":589,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.070903+0000","last_change":"2024-09-13T11:53:06.266346+0000","last_active":"2024-09-13T11:58:48.070903+0000","last_peered":"2024-09-13T11:58:48.070903+0000","last_clean":"2024-09-13T11:58:48.070903+0000","last_became_active":"2024-09-13T11:53:04.258975+0000","last_became_peered":"2024-09-13T11:53:04.258975+0000","last_unstale":"2024-09-13T11:58:48.070903+0000","last_undegraded":"2024-09-13T11:58:48.070903+0000","last_fullsized":"2024-09-13T11:58:48.070903+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:36.773393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055464299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":599,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.972511+0000","last_change":"2024-09-13T11:53:06.266446+0000","last_active":"2024-09-13T11:58:48.972511+0000","last_peered":"2024-09-13T11:58:48.972511+0000","last_clean":"2024-09-13T11:58:48.972511+0000","last_became_active":"2024-09-13T11:53:04.262475+0000","last_became_peered":"2024-09-13T11:53:04.262475+0000","last_unstale":"2024-09-13T11:58:48.972511+0000","last_undegraded":"2024-09-13T11:58:48.972511+0000","last_fullsized":"2024-09-13T11:58:48.972511+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:24:12.639511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055213099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":589,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.068856+0000","last_change":"2024-09-13T11:53:06.266390+0000","last_active":"2024-09-13T11:58:48.068856+0000","last_peered":"2024-09-13T11:58:48.068856+0000","last_clean":"2024-09-13T11:58:48.068856+0000","last_became_active":"2024-09-13T11:53:04.262392+0000","last_became_peered":"2024-09-13T11:53:04.262392+0000","last_unstale":"2024-09-13T11:58:48.068856+0000","last_undegraded":"2024-09-13T11:58:48.068856+0000","last_fullsized":"2024-09-13T11:58:48.068856+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:45.666214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057483700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":587,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.896615+0000","last_change":"2024-09-13T11:53:06.272635+0000","last_active":"2024-09-13T11:58:48.896615+0000","last_peered":"2024-09-13T11:58:48.896615+0000","last_clean":"2024-09-13T11:58:48.896615+0000","last_became_active":"2024-09-13T11:53:04.264659+0000","last_became_peered":"2024-09-13T11:53:04.264659+0000","last_unstale":"2024-09-13T11:58:48.896615+0000","last_undegraded":"2024-09-13T11:58:48.896615+0000","last_fullsized":"2024-09-13T11:58:48.896615+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:53:49.774126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010464509999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":596,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.670015+0000","last_change":"2024-09-13T11:53:06.266298+0000","last_active":"2024-09-13T11:58:48.670015+0000","last_peered":"2024-09-13T11:58:48.670015+0000","last_clean":"2024-09-13T11:58:48.670015+0000","last_became_active":"2024-09-13T11:53:04.270146+0000","last_became_peered":"2024-09-13T11:53:04.270146+0000","last_unstale":"2024-09-13T11:58:48.670015+0000","last_undegraded":"2024-09-13T11:58:48.670015+0000","last_fullsized":"2024-09-13T11:58:48.670015+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:38.439658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061806000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":599,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.972501+0000","last_change":"2024-09-13T11:53:06.266345+0000","last_active":"2024-09-13T11:58:48.972501+0000","last_peered":"2024-09-13T11:58:48.972501+0000","last_clean":"2024-09-13T11:58:48.972501+0000","last_became_active":"2024-09-13T11:53:04.262197+0000","last_became_peered":"2024-09-13T11:53:04.262197+0000","last_unstale":"2024-09-13T11:58:48.972501+0000","last_undegraded":"2024-09-13T11:58:48.972501+0000","last_fullsized":"2024-09-13T11:58:48.972501+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:05:51.189944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044149899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":586,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.896726+0000","last_change":"2024-09-13T11:53:06.272071+0000","last_active":"2024-09-13T11:58:48.896726+0000","last_peered":"2024-09-13T11:58:48.896726+0000","last_clean":"2024-09-13T11:58:48.896726+0000","last_became_active":"2024-09-13T11:53:04.270138+0000","last_became_peered":"2024-09-13T11:53:04.270138+0000","last_unstale":"2024-09-13T11:58:48.896726+0000","last_undegraded":"2024-09-13T11:58:48.896726+0000","last_fullsized":"2024-09-13T11:58:48.896726+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:17.090753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047405,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":691,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.973102+0000","last_change":"2024-09-13T11:52:57.706946+0000","last_active":"2024-09-13T11:58:48.973102+0000","last_peered":"2024-09-13T11:58:48.973102+0000","last_clean":"2024-09-13T11:58:48.973102+0000","last_became_active":"2024-09-13T11:52:57.705894+0000","last_became_peered":"2024-09-13T11:52:57.705894+0000","last_unstale":"2024-09-13T11:58:48.973102+0000","last_undegraded":"2024-09-13T11:58:48.973102+0000","last_fullsized":"2024-09-13T11:58:48.973102+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:52:56.225375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:52:56.225375+0000","last_clean_scrub_stamp":"2024-09-13T11:52:56.225375+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:05.299261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59246,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607629,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31128,"kb_used_data":4100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340712,"statfs":{"total":96636764160,"available":96604889088,"internally_reserved":0,"allocated":4198400,"data_stored":1558821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607628,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35348,"kb_used_data":4160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336492,"statfs":{"total":96636764160,"available":96600567808,"internally_reserved":0,"allocated":4259840,"data_stored":2087749,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607630,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35348,"kb_used_data":4160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336492,"statfs":{"total":96636764160,"available":96600567808,"internally_reserved":0,"allocated":4259840,"data_stored":2087749,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640333,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31136,"kb_used_data":4104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340704,"statfs":{"total":96636764160,"available":96604880896,"internally_reserved":0,"allocated":4202496,"data_stored":1558840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640332,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31132,"kb_used_data":4100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340708,"statfs":{"total":96636764160,"available":96604884992,"internally_reserved":0,"allocated":4198400,"data_stored":1558821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640333,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47772,"kb_used_data":4104,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94324068,"statfs":{"total":96636764160,"available":96587845632,"internally_reserved":0,"allocated":4202496,"data_stored":1558840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:59:05.098 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T11:59:05.098 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:59:05.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.192+0000 7fa403abf640 10 monclient: get_monmap_and_config 2024-09-13T11:59:05.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.192+0000 7fa403abf640 10 monclient: build_initial_monmap 2024-09-13T11:59:05.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.192+0000 7fa403abf640 10 monclient: monmap: 2024-09-13T11:59:05.194 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:05.195 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:05.195 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:05.194202+0000 2024-09-13T11:59:05.195 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:05.194202+0000 2024-09-13T11:59:05.195 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:05.195 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:05.195 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:05.195 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:05.195 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:05.195 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:05.195 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:05.195 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:05.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.192+0000 7fa403abf640 1 Processor -- start 2024-09-13T11:59:05.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.193+0000 7fa403abf640 1 -- start start 2024-09-13T11:59:05.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.193+0000 7fa403abf640 10 monclient: init 2024-09-13T11:59:05.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.193+0000 7fa403abf640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:05.204 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.193+0000 7fa403abf640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-13T11:59:05.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.193+0000 7fa403abf640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3fc0a2fe0 0x7fa3fc09f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:05.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.193+0000 7fa403abf640 10 monclient(hunting): picked mon.noname-b con 0x7fa3fc0a2fe0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:05.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.193+0000 7fa403abf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa3fc133c70 addr v1:172.21.15.122:6789/0 2024-09-13T11:59:05.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.193+0000 7fa403abf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa3fc09c110 addr v1:172.21.15.100:6789/0 2024-09-13T11:59:05.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.193+0000 7fa403abf640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3fc0a2030 con 0x7fa3fc09c110 2024-09-13T11:59:05.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.193+0000 7fa403abf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3fc09dbd0 con 0x7fa3fc133c70 2024-09-13T11:59:05.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.193+0000 7fa403abf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:05.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.193+0000 7fa403abf640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fa3fc0a33c0 con 0x7fa3fc0a2fe0 2024-09-13T11:59:05.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.193+0000 7fa403abf640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:05.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.193+0000 7fa403abf640 10 monclient(hunting): authenticate will time out at 1125.787667s 2024-09-13T11:59:05.205 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.193+0000 7fa40185b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3fc0a2fe0 0x7fa3fc09f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:05.206 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.193+0000 7fa40205c640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fa3fc09c110 0x7fa3fc13bb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:57080/0 (socket says 172.21.15.100:57080) 2024-09-13T11:59:05.206 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.193+0000 7fa40205c640 1 -- 172.21.15.100:0/1637949390 learned_addr learned my addr 172.21.15.100:0/1637949390 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:59:05.206 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.193+0000 7fa40185b640 10 monclient(hunting): get_auth_request con 0x7fa3fc0a2fe0 auth_method 0 2024-09-13T11:59:05.206 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.193+0000 7fa40185b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:05.206 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.193+0000 7fa40185b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:05.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.193+0000 7fa40185b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:05.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.193+0000 7fa40185b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:05.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.193+0000 7fa40185b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:05.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.193+0000 7fa40185b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:05.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa40185b640 10 monclient(hunting): handle_auth_done global_id 24611 payload 274 2024-09-13T11:59:05.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa400859640 1 -- 172.21.15.100:0/1637949390 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1468497693 0 0) 0x7fa3fc0a2030 con 0x7fa3fc09c110 2024-09-13T11:59:05.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa40185b640 1 -- 172.21.15.100:0/1637949390 >> v1:172.21.15.122:6789/0 conn(0x7fa3fc133c70 legacy=0x7fa3fc09ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:05.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa40185b640 1 -- 172.21.15.100:0/1637949390 >> v1:172.21.15.100:6789/0 conn(0x7fa3fc09c110 legacy=0x7fa3fc13bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:05.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa40185b640 10 monclient: _finish_hunting 0 2024-09-13T11:59:05.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa40185b640 1 monclient: found mon.noname-b 2024-09-13T11:59:05.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa40185b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:05.207 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa40185b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:59:05.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa40185b640 1 -- 172.21.15.100:0/1637949390 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3fc09deb0 con 0x7fa3fc0a2fe0 2024-09-13T11:59:05.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa400859640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:59:05.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa40185b640 1 --2- 172.21.15.100:0/1637949390 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3fc0a2fe0 0x7fa3fc09f9a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa3f80097d0 tx=0x7fa3f802a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d97fd0dcc6d6e679 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:05.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa400859640 1 -- 172.21.15.100:0/1637949390 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8037070 con 0x7fa3fc0a2fe0 2024-09-13T11:59:05.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa400859640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:05.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa400859640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:59:05.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa400859640 10 monclient: dump: 2024-09-13T11:59:05.208 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:05.209 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:05.209 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:05.209 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:05.209 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:05.209 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:05.209 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:05.209 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:05.209 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:05.209 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:05.209 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa400859640 10 monclient: _finish_auth 0 2024-09-13T11:59:05.209 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa400859640 10 monclient: _check_auth_tickets 2024-09-13T11:59:05.209 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa400859640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:05.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa400859640 1 -- 172.21.15.100:0/1637949390 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3f802ad90 con 0x7fa3fc0a2fe0 2024-09-13T11:59:05.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa403abf640 5 monclient: authenticate success, global_id 24611 2024-09-13T11:59:05.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa403abf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:05.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa400859640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:05.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa403abf640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:05.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa400859640 1 -- 172.21.15.100:0/1637949390 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8032660 con 0x7fa3fc0a2fe0 2024-09-13T11:59:05.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa403abf640 4 set_mon_vals no callback set 2024-09-13T11:59:05.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa400859640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:05.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa400859640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:59:05.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.194+0000 7fa400859640 10 monclient: dump: 2024-09-13T11:59:05.211 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:05.211 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:05.211 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:05.211 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:05.211 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:05.211 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:05.211 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:05.211 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:05.211 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:05.211 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:05.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.195+0000 7fa403abf640 10 monclient: shutdown 2024-09-13T11:59:05.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.195+0000 7fa403abf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:05.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.195+0000 7fa403abf640 1 -- 172.21.15.100:0/1637949390 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3fc0a2fe0 msgr2=0x7fa3fc09f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:05.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.195+0000 7fa403abf640 1 --2- 172.21.15.100:0/1637949390 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3fc0a2fe0 0x7fa3fc09f9a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa3f80097d0 tx=0x7fa3f802a040 comp rx=0 tx=0).stop 2024-09-13T11:59:05.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.195+0000 7fa403abf640 1 -- 172.21.15.100:0/1637949390 shutdown_connections 2024-09-13T11:59:05.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.195+0000 7fa403abf640 1 --2- 172.21.15.100:0/1637949390 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3fc0a2fe0 0x7fa3fc09f9a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:05.213 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.195+0000 7fa403abf640 1 -- 172.21.15.100:0/1637949390 >> 172.21.15.100:0/1637949390 conn(0x7fa3fc0a6470 msgr2=0x7fa3fc0a2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:05.213 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.195+0000 7fa403abf640 1 -- 172.21.15.100:0/1637949390 shutdown_connections 2024-09-13T11:59:05.213 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.195+0000 7fa403abf640 1 -- 172.21.15.100:0/1637949390 wait complete. 2024-09-13T11:59:05.213 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa403abf640 10 monclient: build_initial_monmap 2024-09-13T11:59:05.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa403abf640 10 monclient: monmap: 2024-09-13T11:59:05.214 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:05.214 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:05.214 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:05.214 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:05.214 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:05.215 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:05.215 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:05.215 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:05.215 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:05.215 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:05.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa403abf640 1 Processor -- start 2024-09-13T11:59:05.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa403abf640 1 -- start start 2024-09-13T11:59:05.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa403abf640 10 monclient: init 2024-09-13T11:59:05.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa403abf640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:05.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa403abf640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:59:05.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa403abf640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3fc09c110 0x7fa3fc11eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:05.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa403abf640 10 monclient(hunting): picked mon.noname-c con 0x7fa3fc09c110 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:05.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa403abf640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3fc0a2fe0 0x7fa3fc11c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:05.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa403abf640 10 monclient(hunting): picked mon.noname-a con 0x7fa3fc0a2fe0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:05.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa403abf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3fc11ccc0 0x7fa3fc11d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:05.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa403abf640 10 monclient(hunting): picked mon.noname-b con 0x7fa3fc11ccc0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:05.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa403abf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:05.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa403abf640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc09ccb0 con 0x7fa3fc0a2fe0 2024-09-13T11:59:05.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa403abf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:05.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa403abf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc09cb30 con 0x7fa3fc11ccc0 2024-09-13T11:59:05.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa403abf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:05.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa403abf640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fa3fc09fee0 con 0x7fa3fc09c110 2024-09-13T11:59:05.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa403abf640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:05.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa403abf640 10 monclient(hunting): authenticate will time out at 1125.790900s 2024-09-13T11:59:05.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa40185b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3fc09c110 0x7fa3fc11eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:05.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa40105a640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3fc0a2fe0 0x7fa3fc11c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:05.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa40185b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3fc09c110 0x7fa3fc11eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:51950/0 (socket says 172.21.15.100:51950) 2024-09-13T11:59:05.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa40185b640 1 -- 172.21.15.100:0/2798798635 learned_addr learned my addr 172.21.15.100:0/2798798635 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:05.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa40185b640 10 monclient(hunting): get_auth_request con 0x7fa3fc09c110 auth_method 0 2024-09-13T11:59:05.220 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa40185b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:05.220 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa40185b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:05.220 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.196+0000 7fa40185b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:05.220 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa40105a640 10 monclient(hunting): get_auth_request con 0x7fa3fc0a2fe0 auth_method 0 2024-09-13T11:59:05.220 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa40105a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:05.220 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa40105a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:05.221 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa40105a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:05.221 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa40105a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:05.221 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa40105a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:05.221 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa40105a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:05.221 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa40185b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:05.222 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa40185b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:05.222 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa40185b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:05.222 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa40105a640 10 monclient(hunting): handle_auth_done global_id 15522 payload 824 2024-09-13T11:59:05.222 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa40105a640 1 -- 172.21.15.100:0/2798798635 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3fc09c110 msgr2=0x7fa3fc11eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:05.222 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa40105a640 1 --2- 172.21.15.100:0/2798798635 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3fc09c110 0x7fa3fc11eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:05.222 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa40105a640 1 -- 172.21.15.100:0/2798798635 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3fc11ccc0 msgr2=0x7fa3fc11d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:05.222 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa40105a640 1 --2- 172.21.15.100:0/2798798635 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3fc11ccc0 0x7fa3fc11d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:05.222 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa40105a640 10 monclient: _finish_hunting 0 2024-09-13T11:59:05.223 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa40105a640 1 monclient: found mon.noname-a 2024-09-13T11:59:05.223 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa40105a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:05.223 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa40105a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:05.223 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa40105a640 1 -- 172.21.15.100:0/2798798635 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3fc076a90 con 0x7fa3fc0a2fe0 2024-09-13T11:59:05.223 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa40105a640 1 --2- 172.21.15.100:0/2798798635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3fc0a2fe0 0x7fa3fc11c780 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa3f000c410 tx=0x7fa3f000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:05.223 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa3ee7fc640 1 -- 172.21.15.100:0/2798798635 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f0017070 con 0x7fa3fc0a2fe0 2024-09-13T11:59:05.223 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa3ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:05.223 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa3ee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:05.223 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa3ee7fc640 10 monclient: dump: 2024-09-13T11:59:05.223 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:05.223 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:05.223 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:05.223 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:05.224 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:05.224 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:05.224 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:05.224 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:05.224 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:05.224 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:05.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa3ee7fc640 10 monclient: _finish_auth 0 2024-09-13T11:59:05.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa3ee7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:59:05.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa3ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:05.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa3ee7fc640 1 -- 172.21.15.100:0/2798798635 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3f0005230 con 0x7fa3fc0a2fe0 2024-09-13T11:59:05.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa403abf640 5 monclient: authenticate success, global_id 15522 2024-09-13T11:59:05.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa403abf640 10 monclient: _renew_subs 2024-09-13T11:59:05.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa403abf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:05.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa403abf640 1 -- 172.21.15.100:0/2798798635 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3fc11db00 con 0x7fa3fc0a2fe0 2024-09-13T11:59:05.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa3ee7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:05.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa3ee7fc640 1 -- 172.21.15.100:0/2798798635 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f00045e0 con 0x7fa3fc0a2fe0 2024-09-13T11:59:05.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa3ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:05.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa3ee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:05.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa3ee7fc640 10 monclient: dump: 2024-09-13T11:59:05.225 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:05.225 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:05.225 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:05.225 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:05.226 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:05.226 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:05.226 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:05.226 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:05.226 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:05.226 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:05.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa3effff640 4 set_mon_vals no callback set 2024-09-13T11:59:05.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa403abf640 10 monclient: _renew_subs 2024-09-13T11:59:05.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa403abf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:05.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.197+0000 7fa403abf640 1 -- 172.21.15.100:0/2798798635 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3fc1205a0 con 0x7fa3fc0a2fe0 2024-09-13T11:59:05.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.198+0000 7fa403abf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:05.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.198+0000 7fa403abf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:05.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.198+0000 7fa403abf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:05.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.198+0000 7fa403abf640 1 -- 172.21.15.100:0/2798798635 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3fc09c580 con 0x7fa3fc0a2fe0 2024-09-13T11:59:05.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.198+0000 7fa3ee7fc640 1 -- 172.21.15.100:0/2798798635 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fa3f001a070 con 0x7fa3fc0a2fe0 2024-09-13T11:59:05.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.198+0000 7fa3ee7fc640 1 --2- 172.21.15.100:0/2798798635 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fa3e003dfa0 0x7fa3e0040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:05.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.198+0000 7fa3ee7fc640 1 -- 172.21.15.100:0/2798798635 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7fa3f0051830 con 0x7fa3fc0a2fe0 2024-09-13T11:59:05.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.198+0000 7fa40185b640 1 --2- 172.21.15.100:0/2798798635 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fa3e003dfa0 0x7fa3e0040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:05.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.198+0000 7fa40185b640 10 monclient: get_auth_request con 0x7fa3e003dfa0 auth_method 0 2024-09-13T11:59:05.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.199+0000 7fa40185b640 1 --2- 172.21.15.100:0/2798798635 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fa3e003dfa0 0x7fa3e0040460 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8000c00 tx=0x7fa3f802e000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:05.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.202+0000 7fa3ee7fc640 1 -- 172.21.15.100:0/2798798635 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fa3f0053d20 con 0x7fa3fc0a2fe0 2024-09-13T11:59:05.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.203+0000 7fa3ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:05.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.203+0000 7fa3ee7fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:59:05.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.373+0000 7fa403abf640 1 -- 172.21.15.100:0/2798798635 --> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa3fc133c70 con 0x7fa3e003dfa0 2024-09-13T11:59:05.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.374+0000 7fa3ee7fc640 1 -- 172.21.15.100:0/2798798635 <== mgr.4114 v2:172.21.15.122:6824/441789830 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34013 (secure 0 0 0) 0x7fa3fc133c70 con 0x7fa3e003dfa0 2024-09-13T11:59:05.377 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:59:05.378 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-13T11:59:05.379 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.376+0000 7fa403abf640 1 -- 172.21.15.100:0/2798798635 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fa3e003dfa0 msgr2=0x7fa3e0040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:05.379 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.376+0000 7fa403abf640 1 --2- 172.21.15.100:0/2798798635 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fa3e003dfa0 0x7fa3e0040460 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8000c00 tx=0x7fa3f802e000 comp rx=0 tx=0).stop 2024-09-13T11:59:05.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.376+0000 7fa403abf640 10 monclient: shutdown 2024-09-13T11:59:05.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.376+0000 7fa403abf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:05.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.376+0000 7fa403abf640 1 -- 172.21.15.100:0/2798798635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3fc0a2fe0 msgr2=0x7fa3fc11c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:05.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.376+0000 7fa403abf640 1 --2- 172.21.15.100:0/2798798635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3fc0a2fe0 0x7fa3fc11c780 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa3f000c410 tx=0x7fa3f000c8d0 comp rx=0 tx=0).stop 2024-09-13T11:59:05.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.377+0000 7fa403abf640 1 -- 172.21.15.100:0/2798798635 shutdown_connections 2024-09-13T11:59:05.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.377+0000 7fa403abf640 1 --2- 172.21.15.100:0/2798798635 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fa3e003dfa0 0x7fa3e0040460 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:05.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.377+0000 7fa403abf640 1 --2- 172.21.15.100:0/2798798635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3fc0a2fe0 0x7fa3fc11c780 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:05.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.377+0000 7fa403abf640 1 --2- 172.21.15.100:0/2798798635 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3fc11ccc0 0x7fa3fc11d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:05.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.377+0000 7fa403abf640 1 --2- 172.21.15.100:0/2798798635 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3fc09c110 0x7fa3fc11eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:05.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.377+0000 7fa403abf640 1 -- 172.21.15.100:0/2798798635 >> 172.21.15.100:0/2798798635 conn(0x7fa3fc0a6470 msgr2=0x7fa3fc0a0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:05.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.377+0000 7fa403abf640 1 -- 172.21.15.100:0/2798798635 shutdown_connections 2024-09-13T11:59:05.381 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.377+0000 7fa403abf640 1 -- 172.21.15.100:0/2798798635 wait complete. 2024-09-13T11:59:05.389 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":493,"stamp":"2024-09-13T11:59:05.286124+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":211864,"kb_used_data":24728,"kb_used_omap":9,"kb_used_meta":186998,"kb_avail":566019176,"statfs":{"total":579820584960,"available":579603636224,"internally_reserved":0,"allocated":25321472,"data_stored":10410820,"data_compressed":59246,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":191486624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002418"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":599,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.972852+0000","last_change":"2024-09-13T11:53:06.266756+0000","last_active":"2024-09-13T11:58:48.972852+0000","last_peered":"2024-09-13T11:58:48.972852+0000","last_clean":"2024-09-13T11:58:48.972852+0000","last_became_active":"2024-09-13T11:53:04.566167+0000","last_became_peered":"2024-09-13T11:53:04.566167+0000","last_unstale":"2024-09-13T11:58:48.972852+0000","last_undegraded":"2024-09-13T11:58:48.972852+0000","last_fullsized":"2024-09-13T11:58:48.972852+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:41.983816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084363899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":589,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.070903+0000","last_change":"2024-09-13T11:53:06.266346+0000","last_active":"2024-09-13T11:58:48.070903+0000","last_peered":"2024-09-13T11:58:48.070903+0000","last_clean":"2024-09-13T11:58:48.070903+0000","last_became_active":"2024-09-13T11:53:04.258975+0000","last_became_peered":"2024-09-13T11:53:04.258975+0000","last_unstale":"2024-09-13T11:58:48.070903+0000","last_undegraded":"2024-09-13T11:58:48.070903+0000","last_fullsized":"2024-09-13T11:58:48.070903+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:36.773393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055464299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":599,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.972511+0000","last_change":"2024-09-13T11:53:06.266446+0000","last_active":"2024-09-13T11:58:48.972511+0000","last_peered":"2024-09-13T11:58:48.972511+0000","last_clean":"2024-09-13T11:58:48.972511+0000","last_became_active":"2024-09-13T11:53:04.262475+0000","last_became_peered":"2024-09-13T11:53:04.262475+0000","last_unstale":"2024-09-13T11:58:48.972511+0000","last_undegraded":"2024-09-13T11:58:48.972511+0000","last_fullsized":"2024-09-13T11:58:48.972511+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:24:12.639511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055213099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":589,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.068856+0000","last_change":"2024-09-13T11:53:06.266390+0000","last_active":"2024-09-13T11:58:48.068856+0000","last_peered":"2024-09-13T11:58:48.068856+0000","last_clean":"2024-09-13T11:58:48.068856+0000","last_became_active":"2024-09-13T11:53:04.262392+0000","last_became_peered":"2024-09-13T11:53:04.262392+0000","last_unstale":"2024-09-13T11:58:48.068856+0000","last_undegraded":"2024-09-13T11:58:48.068856+0000","last_fullsized":"2024-09-13T11:58:48.068856+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:45.666214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057483700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":587,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.896615+0000","last_change":"2024-09-13T11:53:06.272635+0000","last_active":"2024-09-13T11:58:48.896615+0000","last_peered":"2024-09-13T11:58:48.896615+0000","last_clean":"2024-09-13T11:58:48.896615+0000","last_became_active":"2024-09-13T11:53:04.264659+0000","last_became_peered":"2024-09-13T11:53:04.264659+0000","last_unstale":"2024-09-13T11:58:48.896615+0000","last_undegraded":"2024-09-13T11:58:48.896615+0000","last_fullsized":"2024-09-13T11:58:48.896615+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:53:49.774126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010464509999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":596,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.670015+0000","last_change":"2024-09-13T11:53:06.266298+0000","last_active":"2024-09-13T11:58:48.670015+0000","last_peered":"2024-09-13T11:58:48.670015+0000","last_clean":"2024-09-13T11:58:48.670015+0000","last_became_active":"2024-09-13T11:53:04.270146+0000","last_became_peered":"2024-09-13T11:53:04.270146+0000","last_unstale":"2024-09-13T11:58:48.670015+0000","last_undegraded":"2024-09-13T11:58:48.670015+0000","last_fullsized":"2024-09-13T11:58:48.670015+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:38.439658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061806000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":599,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.972501+0000","last_change":"2024-09-13T11:53:06.266345+0000","last_active":"2024-09-13T11:58:48.972501+0000","last_peered":"2024-09-13T11:58:48.972501+0000","last_clean":"2024-09-13T11:58:48.972501+0000","last_became_active":"2024-09-13T11:53:04.262197+0000","last_became_peered":"2024-09-13T11:53:04.262197+0000","last_unstale":"2024-09-13T11:58:48.972501+0000","last_undegraded":"2024-09-13T11:58:48.972501+0000","last_fullsized":"2024-09-13T11:58:48.972501+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:05:51.189944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044149899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":586,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.896726+0000","last_change":"2024-09-13T11:53:06.272071+0000","last_active":"2024-09-13T11:58:48.896726+0000","last_peered":"2024-09-13T11:58:48.896726+0000","last_clean":"2024-09-13T11:58:48.896726+0000","last_became_active":"2024-09-13T11:53:04.270138+0000","last_became_peered":"2024-09-13T11:53:04.270138+0000","last_unstale":"2024-09-13T11:58:48.896726+0000","last_undegraded":"2024-09-13T11:58:48.896726+0000","last_fullsized":"2024-09-13T11:58:48.896726+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:17.090753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047405,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":691,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.973102+0000","last_change":"2024-09-13T11:52:57.706946+0000","last_active":"2024-09-13T11:58:48.973102+0000","last_peered":"2024-09-13T11:58:48.973102+0000","last_clean":"2024-09-13T11:58:48.973102+0000","last_became_active":"2024-09-13T11:52:57.705894+0000","last_became_peered":"2024-09-13T11:52:57.705894+0000","last_unstale":"2024-09-13T11:58:48.973102+0000","last_undegraded":"2024-09-13T11:58:48.973102+0000","last_fullsized":"2024-09-13T11:58:48.973102+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:52:56.225375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:52:56.225375+0000","last_clean_scrub_stamp":"2024-09-13T11:52:56.225375+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:05.299261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59246,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607629,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31128,"kb_used_data":4100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340712,"statfs":{"total":96636764160,"available":96604889088,"internally_reserved":0,"allocated":4198400,"data_stored":1558821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607628,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35348,"kb_used_data":4160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336492,"statfs":{"total":96636764160,"available":96600567808,"internally_reserved":0,"allocated":4259840,"data_stored":2087749,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607630,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35348,"kb_used_data":4160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336492,"statfs":{"total":96636764160,"available":96600567808,"internally_reserved":0,"allocated":4259840,"data_stored":2087749,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640333,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31136,"kb_used_data":4104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340704,"statfs":{"total":96636764160,"available":96604880896,"internally_reserved":0,"allocated":4202496,"data_stored":1558840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640332,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31132,"kb_used_data":4100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340708,"statfs":{"total":96636764160,"available":96604884992,"internally_reserved":0,"allocated":4198400,"data_stored":1558821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640333,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47772,"kb_used_data":4104,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94324068,"statfs":{"total":96636764160,"available":96587845632,"internally_reserved":0,"allocated":4202496,"data_stored":1558840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:59:05.390 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T11:59:05.486 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.484+0000 7fa011209640 10 monclient: get_monmap_and_config 2024-09-13T11:59:05.486 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.484+0000 7fa011209640 10 monclient: build_initial_monmap 2024-09-13T11:59:05.487 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.484+0000 7fa011209640 10 monclient: monmap: 2024-09-13T11:59:05.487 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:05.487 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:05.487 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:05.486393+0000 2024-09-13T11:59:05.487 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:05.486393+0000 2024-09-13T11:59:05.487 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:05.487 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:05.487 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:05.487 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:05.487 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:05.487 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:05.488 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:05.488 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:05.500 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa011209640 1 Processor -- start 2024-09-13T11:59:05.500 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa011209640 1 -- start start 2024-09-13T11:59:05.500 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa011209640 10 monclient: init 2024-09-13T11:59:05.500 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa011209640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:05.500 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa011209640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-13T11:59:05.500 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa011209640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa00c0a15a0 0x7fa00c0a1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:05.501 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa011209640 10 monclient(hunting): picked mon.noname-c con 0x7fa00c0a15a0 addr v2:172.21.15.122:3300/0 2024-09-13T11:59:05.501 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa011209640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa00c09f6d0 addr v1:172.21.15.122:6789/0 2024-09-13T11:59:05.501 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa011209640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa00c0a1ec0 0x7fa00c134ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:05.501 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa011209640 10 monclient(hunting): picked mon.noname-b con 0x7fa00c0a1ec0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:05.501 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa011209640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa00c09ecf0 con 0x7fa00c09f6d0 2024-09-13T11:59:05.501 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa011209640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:05.501 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa011209640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7fa00c09d3b0 con 0x7fa00c0a15a0 2024-09-13T11:59:05.501 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa011209640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:05.501 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa011209640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fa00c09ea10 con 0x7fa00c0a1ec0 2024-09-13T11:59:05.501 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa011209640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:05.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa011209640 10 monclient(hunting): authenticate will time out at 1126.079657s 2024-09-13T11:59:05.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa00b59e640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa00c0a1ec0 0x7fa00c134ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:05.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa00b59e640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa00c0a1ec0 0x7fa00c134ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:51956/0 (socket says 172.21.15.100:51956) 2024-09-13T11:59:05.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa00b59e640 1 -- 172.21.15.100:0/2557116983 learned_addr learned my addr 172.21.15.100:0/2557116983 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:05.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa00b59e640 10 monclient(hunting): get_auth_request con 0x7fa00c0a1ec0 auth_method 0 2024-09-13T11:59:05.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa00b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:05.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa00b59e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:05.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa00b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:05.502 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa00b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:05.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa00b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:05.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa00b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:05.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa00ad9d640 1 --2- 172.21.15.100:0/2557116983 >> v2:172.21.15.122:3300/0 conn(0x7fa00c0a15a0 0x7fa00c0a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:05.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa00ad9d640 10 monclient(hunting): get_auth_request con 0x7fa00c0a15a0 auth_method 0 2024-09-13T11:59:05.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa00ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:05.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa00ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:05.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.485+0000 7fa00ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:05.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa00b59e640 10 monclient(hunting): handle_auth_done global_id 24617 payload 274 2024-09-13T11:59:05.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa00b59e640 1 -- 172.21.15.100:0/2557116983 >> v2:172.21.15.122:3300/0 conn(0x7fa00c0a15a0 msgr2=0x7fa00c0a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:05.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa00b59e640 1 --2- 172.21.15.100:0/2557116983 >> v2:172.21.15.122:3300/0 conn(0x7fa00c0a15a0 0x7fa00c0a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:05.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa00b59e640 1 -- 172.21.15.100:0/2557116983 >> v1:172.21.15.122:6789/0 conn(0x7fa00c09f6d0 legacy=0x7fa00c1343d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:05.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa00b59e640 10 monclient: _finish_hunting 0 2024-09-13T11:59:05.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa00b59e640 1 monclient: found mon.noname-b 2024-09-13T11:59:05.506 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa00b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:05.506 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa00b59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:59:05.506 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa00b59e640 1 -- 172.21.15.100:0/2557116983 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa00c0a2300 con 0x7fa00c0a1ec0 2024-09-13T11:59:05.506 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa00ad9d640 1 --2- 172.21.15.100:0/2557116983 >> v2:172.21.15.122:3300/0 conn(0x7fa00c0a15a0 0x7fa00c0a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:59:05.506 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa00b59e640 1 --2- 172.21.15.100:0/2557116983 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa00c0a1ec0 0x7fa00c134ae0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa00400d150 tx=0x7fa00402c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbc4e67188da85b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:05.506 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa009d9b640 1 -- 172.21.15.100:0/2557116983 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa004039070 con 0x7fa00c0a1ec0 2024-09-13T11:59:05.506 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa009d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:05.506 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa009d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:59:05.506 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa009d9b640 10 monclient: dump: 2024-09-13T11:59:05.506 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:05.507 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:05.507 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:05.507 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:05.507 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:05.507 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:05.507 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:05.507 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:05.507 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:05.507 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:05.507 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa009d9b640 10 monclient: _finish_auth 0 2024-09-13T11:59:05.507 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa009d9b640 10 monclient: _check_auth_tickets 2024-09-13T11:59:05.507 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa009d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:05.508 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa009d9b640 1 -- 172.21.15.100:0/2557116983 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa004009600 con 0x7fa00c0a1ec0 2024-09-13T11:59:05.508 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa011209640 5 monclient: authenticate success, global_id 24617 2024-09-13T11:59:05.508 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa011209640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:05.508 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa009d9b640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:05.508 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa011209640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:05.508 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa009d9b640 1 -- 172.21.15.100:0/2557116983 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0040099f0 con 0x7fa00c0a1ec0 2024-09-13T11:59:05.508 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa009d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:05.508 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa011209640 4 set_mon_vals no callback set 2024-09-13T11:59:05.508 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa009d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:59:05.508 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.486+0000 7fa009d9b640 10 monclient: dump: 2024-09-13T11:59:05.509 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:05.509 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:05.509 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:05.509 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:05.509 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:05.509 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:05.510 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:05.510 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:05.510 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:05.510 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:05.510 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.487+0000 7fa011209640 10 monclient: shutdown 2024-09-13T11:59:05.510 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.487+0000 7fa011209640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:05.510 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.487+0000 7fa011209640 1 -- 172.21.15.100:0/2557116983 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa00c0a1ec0 msgr2=0x7fa00c134ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:05.510 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.487+0000 7fa011209640 1 --2- 172.21.15.100:0/2557116983 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa00c0a1ec0 0x7fa00c134ae0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa00400d150 tx=0x7fa00402c810 comp rx=0 tx=0).stop 2024-09-13T11:59:05.510 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.487+0000 7fa011209640 1 -- 172.21.15.100:0/2557116983 shutdown_connections 2024-09-13T11:59:05.510 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.487+0000 7fa011209640 1 --2- 172.21.15.100:0/2557116983 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa00c0a1ec0 0x7fa00c134ae0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:05.510 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.487+0000 7fa011209640 1 --2- 172.21.15.100:0/2557116983 >> v2:172.21.15.122:3300/0 conn(0x7fa00c0a15a0 0x7fa00c0a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:05.510 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.487+0000 7fa011209640 1 -- 172.21.15.100:0/2557116983 >> 172.21.15.100:0/2557116983 conn(0x7fa00c0a6470 msgr2=0x7fa00c0a2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:05.510 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.487+0000 7fa011209640 1 -- 172.21.15.100:0/2557116983 shutdown_connections 2024-09-13T11:59:05.510 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.487+0000 7fa011209640 1 -- 172.21.15.100:0/2557116983 wait complete. 2024-09-13T11:59:05.511 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.488+0000 7fa011209640 10 monclient: build_initial_monmap 2024-09-13T11:59:05.511 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.488+0000 7fa011209640 10 monclient: monmap: 2024-09-13T11:59:05.511 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:05.511 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:05.512 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:05.512 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:05.512 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:05.512 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:05.512 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:05.512 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:05.513 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:05.513 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:05.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.488+0000 7fa011209640 1 Processor -- start 2024-09-13T11:59:05.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.488+0000 7fa011209640 1 -- start start 2024-09-13T11:59:05.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.488+0000 7fa011209640 10 monclient: init 2024-09-13T11:59:05.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa011209640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:05.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa011209640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:59:05.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa011209640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa00c09f6d0 0x7fa00c1333b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:05.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa011209640 10 monclient(hunting): picked mon.noname-a con 0x7fa00c09f6d0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:05.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa011209640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa00c0a15a0 0x7fa00c1338f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:05.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa011209640 10 monclient(hunting): picked mon.noname-c con 0x7fa00c0a15a0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:05.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa011209640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa00c0a1ec0 0x7fa00c11ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:05.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa011209640 10 monclient(hunting): picked mon.noname-b con 0x7fa00c0a1ec0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:05.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa011209640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:05.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa011209640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fa00c1351a0 con 0x7fa00c09f6d0 2024-09-13T11:59:05.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa011209640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:05.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa011209640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa00c135020 con 0x7fa00c0a1ec0 2024-09-13T11:59:05.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa011209640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:05.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa011209640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fa00c09cfa0 con 0x7fa00c0a15a0 2024-09-13T11:59:05.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa011209640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:05.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa011209640 10 monclient(hunting): authenticate will time out at 1126.083223s 2024-09-13T11:59:05.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00ad9d640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa00c09f6d0 0x7fa00c1333b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:05.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00ad9d640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa00c09f6d0 0x7fa00c1333b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47164/0 (socket says 172.21.15.100:47164) 2024-09-13T11:59:05.514 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00ad9d640 1 -- 172.21.15.100:0/3212692393 learned_addr learned my addr 172.21.15.100:0/3212692393 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:05.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00ad9d640 10 monclient(hunting): get_auth_request con 0x7fa00c09f6d0 auth_method 0 2024-09-13T11:59:05.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00a59c640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa00c0a15a0 0x7fa00c1338f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:05.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:05.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:05.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:05.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00a59c640 10 monclient(hunting): get_auth_request con 0x7fa00c0a15a0 auth_method 0 2024-09-13T11:59:05.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:05.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00a59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:05.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:05.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:05.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:05.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:05.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:05.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:05.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00b59e640 1 --2- 172.21.15.100:0/3212692393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa00c0a1ec0 0x7fa00c11ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:05.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:05.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00b59e640 10 monclient(hunting): get_auth_request con 0x7fa00c0a1ec0 auth_method 0 2024-09-13T11:59:05.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:05.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00b59e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:05.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:05.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:05.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:05.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:05.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00ad9d640 10 monclient(hunting): handle_auth_done global_id 15525 payload 824 2024-09-13T11:59:05.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00ad9d640 1 -- 172.21.15.100:0/3212692393 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa00c0a15a0 msgr2=0x7fa00c1338f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:05.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00ad9d640 1 --2- 172.21.15.100:0/3212692393 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa00c0a15a0 0x7fa00c1338f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:05.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00ad9d640 1 -- 172.21.15.100:0/3212692393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa00c0a1ec0 msgr2=0x7fa00c11ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:05.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00ad9d640 1 --2- 172.21.15.100:0/3212692393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa00c0a1ec0 0x7fa00c11ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:05.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00ad9d640 10 monclient: _finish_hunting 0 2024-09-13T11:59:05.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00ad9d640 1 monclient: found mon.noname-a 2024-09-13T11:59:05.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:05.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:05.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00ad9d640 1 -- 172.21.15.100:0/3212692393 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa00c076a90 con 0x7fa00c09f6d0 2024-09-13T11:59:05.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00a59c640 1 --2- 172.21.15.100:0/3212692393 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa00c0a15a0 0x7fa00c1338f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:59:05.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00ad9d640 1 --2- 172.21.15.100:0/3212692393 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa00c09f6d0 0x7fa00c1333b0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc00e760 tx=0x7f9ffc00ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:05.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7f9ff37fe640 1 -- 172.21.15.100:0/3212692393 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc00f040 con 0x7fa00c09f6d0 2024-09-13T11:59:05.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7f9ff37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:05.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7f9ff37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:05.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7f9ff37fe640 10 monclient: dump: 2024-09-13T11:59:05.518 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:05.518 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:05.518 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:05.518 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:05.518 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:05.518 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:05.518 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:05.518 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:05.519 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:05.519 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:05.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7f9ff37fe640 10 monclient: _finish_auth 0 2024-09-13T11:59:05.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7f9ff37fe640 10 monclient: _check_auth_tickets 2024-09-13T11:59:05.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7f9ff37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:05.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7f9ff37fe640 1 -- 172.21.15.100:0/3212692393 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ffc004230 con 0x7fa00c09f6d0 2024-09-13T11:59:05.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa011209640 5 monclient: authenticate success, global_id 15525 2024-09-13T11:59:05.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa011209640 10 monclient: _renew_subs 2024-09-13T11:59:05.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa011209640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:05.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa011209640 1 -- 172.21.15.100:0/3212692393 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa00c123da0 con 0x7fa00c09f6d0 2024-09-13T11:59:05.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7f9ff37fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:05.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7f9ff37fe640 1 -- 172.21.15.100:0/3212692393 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc00ee70 con 0x7fa00c09f6d0 2024-09-13T11:59:05.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7f9ff37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:05.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.490+0000 7f9ff37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:05.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.490+0000 7f9ff37fe640 10 monclient: dump: 2024-09-13T11:59:05.520 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:05.520 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:05.520 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:05.520 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:05.520 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:05.520 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:05.520 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:05.520 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:05.520 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:05.520 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:05.520 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.490+0000 7fa011209640 10 monclient: _renew_subs 2024-09-13T11:59:05.520 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.489+0000 7fa00959a640 4 set_mon_vals no callback set 2024-09-13T11:59:05.520 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.490+0000 7fa011209640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:05.520 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.490+0000 7fa011209640 1 -- 172.21.15.100:0/3212692393 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa00c123500 con 0x7fa00c09f6d0 2024-09-13T11:59:05.520 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.490+0000 7f9ff37fe640 1 -- 172.21.15.100:0/3212692393 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f9ffc00c670 con 0x7fa00c09f6d0 2024-09-13T11:59:05.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.491+0000 7f9ff37fe640 1 --2- 172.21.15.100:0/3212692393 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9ff803dfa0 0x7f9ff8040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:05.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.491+0000 7fa011209640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:05.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.491+0000 7fa011209640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:05.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.491+0000 7fa011209640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:05.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.491+0000 7fa011209640 1 -- 172.21.15.100:0/3212692393 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa00c134e40 con 0x7fa00c09f6d0 2024-09-13T11:59:05.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.491+0000 7f9ff37fe640 1 -- 172.21.15.100:0/3212692393 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7f9ffc0523c0 con 0x7fa00c09f6d0 2024-09-13T11:59:05.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.491+0000 7fa00a59c640 1 --2- 172.21.15.100:0/3212692393 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9ff803dfa0 0x7f9ff8040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:05.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.491+0000 7fa00a59c640 10 monclient: get_auth_request con 0x7f9ff803dfa0 auth_method 0 2024-09-13T11:59:05.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.491+0000 7fa00a59c640 1 --2- 172.21.15.100:0/3212692393 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9ff803dfa0 0x7f9ff8040460 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa00c123430 tx=0x7fa000009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:05.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.495+0000 7f9ff37fe640 1 -- 172.21.15.100:0/3212692393 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fa00c134e40 con 0x7fa00c09f6d0 2024-09-13T11:59:05.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.495+0000 7f9ff37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:05.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.495+0000 7f9ff37fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:59:05.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.670+0000 7fa011209640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:59:05.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.670+0000 7fa011209640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:59:05.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.670+0000 7fa011209640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:05.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.670+0000 7fa011209640 1 -- 172.21.15.100:0/3212692393 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa00c0a1980 con 0x7fa00c09f6d0 2024-09-13T11:59:05.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.670+0000 7f9ff37fe640 1 -- 172.21.15.100:0/3212692393 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v313) ==== 74+0+12326 (secure 0 0 0) 0x7fa00c076a90 con 0x7fa00c09f6d0 2024-09-13T11:59:05.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.670+0000 7f9ff37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:59:05.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.670+0000 7f9ff37fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:59:05.673 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:59:05.674 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":313,"fsid":"0b1f4b4d-fad4-4dd3-86ac-a286d80846c0","created":"2024-09-13T11:52:46.158794+0000","modified":"2024-09-13T11:58:48.062956+0000","last_up_change":"2024-09-13T11:52:56.225375+0000","last_in_change":"2024-09-13T11:52:47.800207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":100,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T11:52:55.233767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T11:53:03.003491+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac784459-dbff-42ed-941e-70c741392dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":311,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":92840425},{"type":"v1","addr":"172.21.15.100:6817","nonce":92840425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":92840425},{"type":"v1","addr":"172.21.15.100:6819","nonce":92840425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6822","nonce":92840425},{"type":"v1","addr":"172.21.15.100:6823","nonce":92840425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":92840425},{"type":"v1","addr":"172.21.15.100:6821","nonce":92840425}]},"public_addr":"172.21.15.100:6817/92840425","cluster_addr":"172.21.15.100:6819/92840425","heartbeat_back_addr":"172.21.15.100:6823/92840425","heartbeat_front_addr":"172.21.15.100:6821/92840425","state":["exists","up"]},{"osd":1,"uuid":"8d8d39e7-e8f1-499a-a01b-e7ba49844eae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":311,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":1385319817},{"type":"v1","addr":"172.21.15.100:6809","nonce":1385319817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":1385319817},{"type":"v1","addr":"172.21.15.100:6811","nonce":1385319817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":1385319817},{"type":"v1","addr":"172.21.15.100:6815","nonce":1385319817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":1385319817},{"type":"v1","addr":"172.21.15.100:6813","nonce":1385319817}]},"public_addr":"172.21.15.100:6809/1385319817","cluster_addr":"172.21.15.100:6811/1385319817","heartbeat_back_addr":"172.21.15.100:6815/1385319817","heartbeat_front_addr":"172.21.15.100:6813/1385319817","state":["exists","up"]},{"osd":2,"uuid":"e0bd3e30-0ff3-4067-83d3-ef8d84b29097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":311,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":2354314235},{"type":"v1","addr":"172.21.15.100:6801","nonce":2354314235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":2354314235},{"type":"v1","addr":"172.21.15.100:6803","nonce":2354314235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":2354314235},{"type":"v1","addr":"172.21.15.100:6807","nonce":2354314235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":2354314235},{"type":"v1","addr":"172.21.15.100:6805","nonce":2354314235}]},"public_addr":"172.21.15.100:6801/2354314235","cluster_addr":"172.21.15.100:6803/2354314235","heartbeat_back_addr":"172.21.15.100:6807/2354314235","heartbeat_front_addr":"172.21.15.100:6805/2354314235","state":["exists","up"]},{"osd":3,"uuid":"eca4429b-38cd-488a-b31d-c5856d701bde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":311,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3206511700},{"type":"v1","addr":"172.21.15.122:6809","nonce":3206511700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3206511700},{"type":"v1","addr":"172.21.15.122:6811","nonce":3206511700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3206511700},{"type":"v1","addr":"172.21.15.122:6815","nonce":3206511700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3206511700},{"type":"v1","addr":"172.21.15.122:6813","nonce":3206511700}]},"public_addr":"172.21.15.122:6809/3206511700","cluster_addr":"172.21.15.122:6811/3206511700","heartbeat_back_addr":"172.21.15.122:6815/3206511700","heartbeat_front_addr":"172.21.15.122:6813/3206511700","state":["exists","up"]},{"osd":4,"uuid":"b7ddc8f1-38d5-496a-9bf7-181eab7e5516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":311,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":777240702},{"type":"v1","addr":"172.21.15.122:6817","nonce":777240702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":777240702},{"type":"v1","addr":"172.21.15.122:6819","nonce":777240702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":777240702},{"type":"v1","addr":"172.21.15.122:6823","nonce":777240702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":777240702},{"type":"v1","addr":"172.21.15.122:6821","nonce":777240702}]},"public_addr":"172.21.15.122:6817/777240702","cluster_addr":"172.21.15.122:6819/777240702","heartbeat_back_addr":"172.21.15.122:6823/777240702","heartbeat_front_addr":"172.21.15.122:6821/777240702","state":["exists","up"]},{"osd":5,"uuid":"9010f8e0-c97b-451c-8b29-3ccb99e982a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":311,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":904725860},{"type":"v1","addr":"172.21.15.122:6801","nonce":904725860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":904725860},{"type":"v1","addr":"172.21.15.122:6803","nonce":904725860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":904725860},{"type":"v1","addr":"172.21.15.122:6807","nonce":904725860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":904725860},{"type":"v1","addr":"172.21.15.122:6805","nonce":904725860}]},"public_addr":"172.21.15.122:6801/904725860","cluster_addr":"172.21.15.122:6803/904725860","heartbeat_back_addr":"172.21.15.122:6807/904725860","heartbeat_front_addr":"172.21.15.122:6805/904725860","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:54.752725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:53.791663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:53.535397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:54.348980+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:54.658725+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:52:53.932256+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:59:05.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.673+0000 7fa011209640 1 -- 172.21.15.100:0/3212692393 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9ff803dfa0 msgr2=0x7f9ff8040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:05.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.673+0000 7fa011209640 1 --2- 172.21.15.100:0/3212692393 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9ff803dfa0 0x7f9ff8040460 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa00c123430 tx=0x7fa000009040 comp rx=0 tx=0).stop 2024-09-13T11:59:05.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.673+0000 7fa011209640 10 monclient: shutdown 2024-09-13T11:59:05.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.673+0000 7fa011209640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:05.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.673+0000 7fa011209640 1 -- 172.21.15.100:0/3212692393 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa00c09f6d0 msgr2=0x7fa00c1333b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:05.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.673+0000 7fa011209640 1 --2- 172.21.15.100:0/3212692393 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa00c09f6d0 0x7fa00c1333b0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc00e760 tx=0x7f9ffc00ec20 comp rx=0 tx=0).stop 2024-09-13T11:59:05.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.673+0000 7fa011209640 1 -- 172.21.15.100:0/3212692393 shutdown_connections 2024-09-13T11:59:05.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.673+0000 7fa011209640 1 --2- 172.21.15.100:0/3212692393 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f9ff803dfa0 0x7f9ff8040460 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:05.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.673+0000 7fa011209640 1 --2- 172.21.15.100:0/3212692393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa00c0a1ec0 0x7fa00c11ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:05.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.673+0000 7fa011209640 1 --2- 172.21.15.100:0/3212692393 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa00c0a15a0 0x7fa00c1338f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:05.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.673+0000 7fa011209640 1 --2- 172.21.15.100:0/3212692393 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa00c09f6d0 0x7fa00c1333b0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:05.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.673+0000 7fa011209640 1 -- 172.21.15.100:0/3212692393 >> 172.21.15.100:0/3212692393 conn(0x7fa00c0a6470 msgr2=0x7fa00c0a41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:05.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.673+0000 7fa011209640 1 -- 172.21.15.100:0/3212692393 shutdown_connections 2024-09-13T11:59:05.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:05.673+0000 7fa011209640 1 -- 172.21.15.100:0/3212692393 wait complete. 2024-09-13T11:59:06.686 INFO:tasks.ceph:Scrubbing osd.0 2024-09-13T11:59:06.687 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:59:06.800 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.798+0000 7fdaa4abe640 10 monclient: get_monmap_and_config 2024-09-13T11:59:06.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.798+0000 7fdaa4abe640 10 monclient: build_initial_monmap 2024-09-13T11:59:06.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.798+0000 7fdaa4abe640 10 monclient: monmap: 2024-09-13T11:59:06.801 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:06.801 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:06.801 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:06.800560+0000 2024-09-13T11:59:06.801 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:06.800560+0000 2024-09-13T11:59:06.801 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:06.801 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:06.801 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:06.801 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:06.801 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:06.801 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:06.801 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:06.802 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:06.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.799+0000 7fdaa4abe640 1 Processor -- start 2024-09-13T11:59:06.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.799+0000 7fdaa4abe640 1 -- start start 2024-09-13T11:59:06.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.799+0000 7fdaa4abe640 10 monclient: init 2024-09-13T11:59:06.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.799+0000 7fdaa4abe640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:06.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.799+0000 7fdaa4abe640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-13T11:59:06.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.799+0000 7fdaa4abe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdaa00a37f0 addr v1:172.21.15.122:6789/0 2024-09-13T11:59:06.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.799+0000 7fdaa4abe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdaa009f6b0 0x7fdaa013a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:06.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.799+0000 7fdaa4abe640 10 monclient(hunting): picked mon.noname-b con 0x7fdaa009f6b0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:06.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.799+0000 7fdaa4abe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fdaa00a42e0 0x7fdaa013cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:06.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.799+0000 7fdaa4abe640 10 monclient(hunting): picked mon.noname-c con 0x7fdaa00a42e0 addr v2:172.21.15.122:3300/0 2024-09-13T11:59:06.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.799+0000 7fdaa4abe640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdaa009fb30 con 0x7fdaa00a37f0 2024-09-13T11:59:06.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.799+0000 7fdaa4abe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:06.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.799+0000 7fdaa4abe640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7fdaa00a1470 con 0x7fdaa00a42e0 2024-09-13T11:59:06.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.799+0000 7fdaa4abe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:06.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.799+0000 7fdaa4abe640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fdaa00a1e30 con 0x7fdaa009f6b0 2024-09-13T11:59:06.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.799+0000 7fdaa4abe640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:06.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.799+0000 7fdaa4abe640 10 monclient(hunting): authenticate will time out at 1127.393971s 2024-09-13T11:59:06.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda95d9b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdaa009f6b0 0x7fdaa013a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:06.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda95d9b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdaa009f6b0 0x7fdaa013a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:51974/0 (socket says 172.21.15.100:51974) 2024-09-13T11:59:06.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda95d9b640 1 -- 172.21.15.100:0/2021556721 learned_addr learned my addr 172.21.15.100:0/2021556721 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:06.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda95d9b640 10 monclient(hunting): get_auth_request con 0x7fdaa009f6b0 auth_method 0 2024-09-13T11:59:06.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda95d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:06.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda95d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:06.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda95d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:06.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda95d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:06.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda95d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:06.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda95d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:06.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda9ed9d640 1 --2- 172.21.15.100:0/2021556721 >> v2:172.21.15.122:3300/0 conn(0x7fdaa00a42e0 0x7fdaa013cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:06.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda9ed9d640 10 monclient(hunting): get_auth_request con 0x7fdaa00a42e0 auth_method 0 2024-09-13T11:59:06.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda9ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:06.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda9ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:06.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda9ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:06.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda95d9b640 10 monclient(hunting): handle_auth_done global_id 24623 payload 274 2024-09-13T11:59:06.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda95d9b640 1 -- 172.21.15.100:0/2021556721 >> v2:172.21.15.122:3300/0 conn(0x7fdaa00a42e0 msgr2=0x7fdaa013cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:06.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda95d9b640 1 --2- 172.21.15.100:0/2021556721 >> v2:172.21.15.122:3300/0 conn(0x7fdaa00a42e0 0x7fdaa013cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:06.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda95d9b640 1 -- 172.21.15.100:0/2021556721 >> v1:172.21.15.122:6789/0 conn(0x7fdaa00a37f0 legacy=0x7fdaa00a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:06.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda95d9b640 10 monclient: _finish_hunting 0 2024-09-13T11:59:06.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda95d9b640 1 monclient: found mon.noname-b 2024-09-13T11:59:06.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda95d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:06.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda95d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:59:06.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda95d9b640 1 -- 172.21.15.100:0/2021556721 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa00a0bd0 con 0x7fdaa009f6b0 2024-09-13T11:59:06.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.800+0000 7fda9ed9d640 1 --2- 172.21.15.100:0/2021556721 >> v2:172.21.15.122:3300/0 conn(0x7fdaa00a42e0 0x7fdaa013cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:59:06.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fda95d9b640 1 --2- 172.21.15.100:0/2021556721 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdaa009f6b0 0x7fdaa013a850 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fda88009870 tx=0x7fda8802a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b1528e6680d8ca0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:06.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fda9dd9b640 1 -- 172.21.15.100:0/2021556721 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda88037070 con 0x7fdaa009f6b0 2024-09-13T11:59:06.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fda9dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:06.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fda9dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:59:06.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fda9dd9b640 10 monclient: dump: 2024-09-13T11:59:06.814 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:06.814 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:06.814 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:06.814 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:06.814 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:06.814 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:06.814 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:06.815 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:06.815 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:06.815 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:06.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fda9dd9b640 10 monclient: _finish_auth 0 2024-09-13T11:59:06.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fda9dd9b640 10 monclient: _check_auth_tickets 2024-09-13T11:59:06.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fda9dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:06.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fda9dd9b640 1 -- 172.21.15.100:0/2021556721 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda88032440 con 0x7fdaa009f6b0 2024-09-13T11:59:06.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fdaa4abe640 5 monclient: authenticate success, global_id 24623 2024-09-13T11:59:06.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fdaa4abe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:06.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fda9dd9b640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:06.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fdaa4abe640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:06.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fda9dd9b640 1 -- 172.21.15.100:0/2021556721 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda88032810 con 0x7fdaa009f6b0 2024-09-13T11:59:06.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fda9dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:06.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fdaa4abe640 4 set_mon_vals no callback set 2024-09-13T11:59:06.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fda9dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:59:06.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fda9dd9b640 10 monclient: dump: 2024-09-13T11:59:06.816 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:06.816 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:06.816 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:06.816 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:06.816 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:06.816 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:06.816 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:06.816 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:06.816 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:06.816 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:06.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fdaa4abe640 10 monclient: shutdown 2024-09-13T11:59:06.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fdaa4abe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:06.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fdaa4abe640 1 -- 172.21.15.100:0/2021556721 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdaa009f6b0 msgr2=0x7fdaa013a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:06.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fdaa4abe640 1 --2- 172.21.15.100:0/2021556721 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdaa009f6b0 0x7fdaa013a850 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fda88009870 tx=0x7fda8802a130 comp rx=0 tx=0).stop 2024-09-13T11:59:06.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fdaa4abe640 1 -- 172.21.15.100:0/2021556721 shutdown_connections 2024-09-13T11:59:06.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fdaa4abe640 1 --2- 172.21.15.100:0/2021556721 >> v2:172.21.15.122:3300/0 conn(0x7fdaa00a42e0 0x7fdaa013cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:06.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fdaa4abe640 1 --2- 172.21.15.100:0/2021556721 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdaa009f6b0 0x7fdaa013a850 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:06.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fdaa4abe640 1 -- 172.21.15.100:0/2021556721 >> 172.21.15.100:0/2021556721 conn(0x7fdaa00a8480 msgr2=0x7fdaa00a4950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:06.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fdaa4abe640 1 -- 172.21.15.100:0/2021556721 shutdown_connections 2024-09-13T11:59:06.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.801+0000 7fdaa4abe640 1 -- 172.21.15.100:0/2021556721 wait complete. 2024-09-13T11:59:06.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.802+0000 7fdaa4abe640 10 monclient: build_initial_monmap 2024-09-13T11:59:06.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.802+0000 7fdaa4abe640 10 monclient: monmap: 2024-09-13T11:59:06.817 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:06.818 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:06.818 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:06.818 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:06.818 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:06.818 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:06.818 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:06.818 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:06.818 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:06.818 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:06.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fdaa4abe640 1 Processor -- start 2024-09-13T11:59:06.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fdaa4abe640 1 -- start start 2024-09-13T11:59:06.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fdaa4abe640 10 monclient: init 2024-09-13T11:59:06.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fdaa4abe640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:06.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fdaa4abe640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:59:06.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fdaa4abe640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdaa009f6b0 0x7fdaa0135760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:06.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fdaa4abe640 10 monclient(hunting): picked mon.noname-c con 0x7fdaa009f6b0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:06.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fdaa4abe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdaa00a37f0 0x7fdaa0135ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:06.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fdaa4abe640 10 monclient(hunting): picked mon.noname-b con 0x7fdaa00a37f0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:06.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fdaa4abe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdaa00a42e0 0x7fdaa011e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:06.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fdaa4abe640 10 monclient(hunting): picked mon.noname-a con 0x7fdaa00a42e0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:06.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fdaa4abe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:06.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fdaa4abe640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa009ef00 con 0x7fdaa00a42e0 2024-09-13T11:59:06.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fdaa4abe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:06.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fdaa4abe640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa013d2e0 con 0x7fdaa00a37f0 2024-09-13T11:59:06.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fdaa4abe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:06.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fdaa4abe640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fdaa01cd680 con 0x7fdaa009f6b0 2024-09-13T11:59:06.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fdaa4abe640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:06.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fdaa4abe640 10 monclient(hunting): authenticate will time out at 1127.397650s 2024-09-13T11:59:06.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda9e59c640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdaa009f6b0 0x7fdaa0135760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:06.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda9ed9d640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdaa00a42e0 0x7fdaa011e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:06.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda9e59c640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdaa009f6b0 0x7fdaa0135760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:51988/0 (socket says 172.21.15.100:51988) 2024-09-13T11:59:06.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda9e59c640 1 -- 172.21.15.100:0/4189067730 learned_addr learned my addr 172.21.15.100:0/4189067730 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:06.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda9e59c640 10 monclient(hunting): get_auth_request con 0x7fdaa009f6b0 auth_method 0 2024-09-13T11:59:06.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda9e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:06.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda9e59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:06.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda9e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:06.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda9ed9d640 10 monclient(hunting): get_auth_request con 0x7fdaa00a42e0 auth_method 0 2024-09-13T11:59:06.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda9ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:06.820 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda9ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:06.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda9ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:06.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda95d9b640 1 --2- 172.21.15.100:0/4189067730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdaa00a37f0 0x7fdaa0135ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:06.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda95d9b640 10 monclient(hunting): get_auth_request con 0x7fdaa00a37f0 auth_method 0 2024-09-13T11:59:06.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda95d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:06.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda95d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:06.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda95d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:06.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda9ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:06.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda9ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:06.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda9ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:06.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda95d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:06.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda95d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:06.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda95d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:06.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda9e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:06.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda9e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:06.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.803+0000 7fda9e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:06.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda9ed9d640 10 monclient(hunting): handle_auth_done global_id 15528 payload 824 2024-09-13T11:59:06.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda9ed9d640 1 -- 172.21.15.100:0/4189067730 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdaa009f6b0 msgr2=0x7fdaa0135760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:06.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda9ed9d640 1 --2- 172.21.15.100:0/4189067730 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdaa009f6b0 0x7fdaa0135760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:06.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda9ed9d640 1 -- 172.21.15.100:0/4189067730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdaa00a37f0 msgr2=0x7fdaa0135ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:06.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda9ed9d640 1 --2- 172.21.15.100:0/4189067730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdaa00a37f0 0x7fdaa0135ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:06.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda9ed9d640 10 monclient: _finish_hunting 0 2024-09-13T11:59:06.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda9ed9d640 1 monclient: found mon.noname-a 2024-09-13T11:59:06.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda9ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:06.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda9ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:06.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda9ed9d640 1 -- 172.21.15.100:0/4189067730 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa00711b0 con 0x7fdaa00a42e0 2024-09-13T11:59:06.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda95d9b640 1 --2- 172.21.15.100:0/4189067730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdaa00a37f0 0x7fdaa0135ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:59:06.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda9e59c640 1 --2- 172.21.15.100:0/4189067730 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdaa009f6b0 0x7fdaa0135760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:59:06.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda9ed9d640 1 --2- 172.21.15.100:0/4189067730 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdaa00a42e0 0x7fdaa011e4a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fda98007ef0 tx=0x7fda9800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:06.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda977fe640 1 -- 172.21.15.100:0/4189067730 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9801a070 con 0x7fdaa00a42e0 2024-09-13T11:59:06.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:06.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:06.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda977fe640 10 monclient: dump: 2024-09-13T11:59:06.823 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:06.823 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:06.823 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:06.823 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:06.823 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:06.823 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:06.823 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:06.823 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:06.824 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:06.824 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:06.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda977fe640 10 monclient: _finish_auth 0 2024-09-13T11:59:06.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda977fe640 10 monclient: _check_auth_tickets 2024-09-13T11:59:06.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:06.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fdaa4abe640 5 monclient: authenticate success, global_id 15528 2024-09-13T11:59:06.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda977fe640 1 -- 172.21.15.100:0/4189067730 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda98003b90 con 0x7fdaa00a42e0 2024-09-13T11:59:06.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fdaa4abe640 10 monclient: _renew_subs 2024-09-13T11:59:06.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fdaa4abe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:06.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fdaa4abe640 1 -- 172.21.15.100:0/4189067730 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaa009f0b0 con 0x7fdaa00a42e0 2024-09-13T11:59:06.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda977fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:06.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda977fe640 1 -- 172.21.15.100:0/4189067730 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda98004030 con 0x7fdaa00a42e0 2024-09-13T11:59:06.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:06.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:06.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda977fe640 10 monclient: dump: 2024-09-13T11:59:06.825 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:06.825 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:06.825 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:06.825 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:06.825 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:06.825 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:06.825 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:06.825 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:06.825 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:06.825 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:06.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fdaa4abe640 10 monclient: _renew_subs 2024-09-13T11:59:06.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fdaa4abe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:06.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fdaa4abe640 1 -- 172.21.15.100:0/4189067730 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaa0122210 con 0x7fdaa00a42e0 2024-09-13T11:59:06.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.804+0000 7fda9d59a640 4 set_mon_vals no callback set 2024-09-13T11:59:06.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.805+0000 7fdaa4abe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:06.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.805+0000 7fdaa4abe640 1 -- 172.21.15.100:0/4189067730 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdaa00a6e20 con 0x7fdaa00a42e0 2024-09-13T11:59:06.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.805+0000 7fda977fe640 1 -- 172.21.15.100:0/4189067730 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fda9801d070 con 0x7fdaa00a42e0 2024-09-13T11:59:06.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.805+0000 7fda977fe640 1 --2- 172.21.15.100:0/4189067730 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fda8003dfa0 0x7fda80040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:06.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.805+0000 7fda977fe640 1 -- 172.21.15.100:0/4189067730 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7fda980528d0 con 0x7fdaa00a42e0 2024-09-13T11:59:06.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.805+0000 7fda977fe640 1 --2- 172.21.15.100:0/4189067730 >> [v2:172.21.15.100:6816/92840425,v1:172.21.15.100:6817/92840425] conn(0x7fda80043da0 0x7fda800461c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:06.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.805+0000 7fda977fe640 1 -- 172.21.15.100:0/4189067730 --> [v2:172.21.15.100:6816/92840425,v1:172.21.15.100:6817/92840425] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fda9801de50 con 0x7fda80043da0 2024-09-13T11:59:06.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.805+0000 7fda977fe640 1 -- 172.21.15.100:0/4189067730 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_get_version_reply(handle=1 version=313) ==== 24+0+0 (secure 0 0 0) 0x7fda980572c0 con 0x7fdaa00a42e0 2024-09-13T11:59:06.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.805+0000 7fda977fe640 10 monclient: handle_get_version_reply finishing 1 version 313 2024-09-13T11:59:06.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.805+0000 7fda9e59c640 1 --2- 172.21.15.100:0/4189067730 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fda8003dfa0 0x7fda80040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:06.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.805+0000 7fda9e59c640 10 monclient: get_auth_request con 0x7fda8003dfa0 auth_method 0 2024-09-13T11:59:06.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.805+0000 7fda95d9b640 1 --2- 172.21.15.100:0/4189067730 >> [v2:172.21.15.100:6816/92840425,v1:172.21.15.100:6817/92840425] conn(0x7fda80043da0 0x7fda800461c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:06.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.806+0000 7fda95d9b640 10 monclient: get_auth_request con 0x7fda80043da0 auth_method 0 2024-09-13T11:59:06.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.806+0000 7fda95d9b640 1 --2- 172.21.15.100:0/4189067730 >> [v2:172.21.15.100:6816/92840425,v1:172.21.15.100:6817/92840425] conn(0x7fda80043da0 0x7fda800461c0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:06.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.806+0000 7fda9e59c640 1 --2- 172.21.15.100:0/4189067730 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fda8003dfa0 0x7fda80040460 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fda90000f60 tx=0x7fda90009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:06.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.806+0000 7fda977fe640 1 -- 172.21.15.100:0/4189067730 <== osd.0 v2:172.21.15.100:6816/92840425 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fda9801de50 con 0x7fda80043da0 2024-09-13T11:59:06.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.818+0000 7fdaa4abe640 1 -- 172.21.15.100:0/4189067730 --> [v2:172.21.15.100:6816/92840425,v1:172.21.15.100:6817/92840425] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fdaa00a3bd0 con 0x7fda80043da0 2024-09-13T11:59:06.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.819+0000 7fda977fe640 1 -- 172.21.15.100:0/4189067730 <== osd.0 v2:172.21.15.100:6816/92840425 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fdaa00a3bd0 con 0x7fda80043da0 2024-09-13T11:59:06.828 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:59:06.828 INFO:teuthology.orchestra.run.smithi100.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:59:06.828 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:59:06.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.819+0000 7fdaa4abe640 1 -- 172.21.15.100:0/4189067730 >> [v2:172.21.15.100:6816/92840425,v1:172.21.15.100:6817/92840425] conn(0x7fda80043da0 msgr2=0x7fda800461c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:06.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.819+0000 7fdaa4abe640 1 --2- 172.21.15.100:0/4189067730 >> [v2:172.21.15.100:6816/92840425,v1:172.21.15.100:6817/92840425] conn(0x7fda80043da0 0x7fda800461c0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:06.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.819+0000 7fdaa4abe640 1 -- 172.21.15.100:0/4189067730 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fda8003dfa0 msgr2=0x7fda80040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:06.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.819+0000 7fdaa4abe640 1 --2- 172.21.15.100:0/4189067730 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fda8003dfa0 0x7fda80040460 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fda90000f60 tx=0x7fda90009040 comp rx=0 tx=0).stop 2024-09-13T11:59:06.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.819+0000 7fdaa4abe640 10 monclient: shutdown 2024-09-13T11:59:06.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.819+0000 7fdaa4abe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:06.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.819+0000 7fdaa4abe640 1 -- 172.21.15.100:0/4189067730 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdaa00a42e0 msgr2=0x7fdaa011e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:06.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.819+0000 7fdaa4abe640 1 --2- 172.21.15.100:0/4189067730 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdaa00a42e0 0x7fdaa011e4a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fda98007ef0 tx=0x7fda9800d4a0 comp rx=0 tx=0).stop 2024-09-13T11:59:06.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.819+0000 7fda9ed9d640 1 -- 172.21.15.100:0/4189067730 reap_dead start 2024-09-13T11:59:06.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.819+0000 7fdaa4abe640 1 -- 172.21.15.100:0/4189067730 shutdown_connections 2024-09-13T11:59:06.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.819+0000 7fdaa4abe640 1 -- 172.21.15.100:0/4189067730 >> 172.21.15.100:0/4189067730 conn(0x7fdaa00a8480 msgr2=0x7fdaa00a6630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:06.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.819+0000 7fdaa4abe640 1 -- 172.21.15.100:0/4189067730 shutdown_connections 2024-09-13T11:59:06.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.819+0000 7fdaa4abe640 1 -- 172.21.15.100:0/4189067730 wait complete. 2024-09-13T11:59:06.832 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-13T11:59:06.929 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.926+0000 7fc5f2944640 10 monclient: get_monmap_and_config 2024-09-13T11:59:06.929 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.926+0000 7fc5f2944640 10 monclient: build_initial_monmap 2024-09-13T11:59:06.929 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.926+0000 7fc5f2944640 10 monclient: monmap: 2024-09-13T11:59:06.929 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:06.929 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:06.929 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:06.928702+0000 2024-09-13T11:59:06.929 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:06.928702+0000 2024-09-13T11:59:06.929 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:06.929 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:06.929 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:06.929 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:06.930 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:06.930 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:06.930 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:06.930 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:06.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.927+0000 7fc5f2944640 1 Processor -- start 2024-09-13T11:59:06.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.927+0000 7fc5f2944640 1 -- start start 2024-09-13T11:59:06.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.927+0000 7fc5f2944640 10 monclient: init 2024-09-13T11:59:06.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f2944640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:06.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f2944640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-13T11:59:06.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f2944640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc5ec0a37d0 0x7fc5ec0a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:06.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f2944640 10 monclient(hunting): picked mon.noname-a con 0x7fc5ec0a37d0 addr v2:172.21.15.100:3300/0 2024-09-13T11:59:06.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f2944640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc5ec06eeb0 addr v1:172.21.15.100:6789/0 2024-09-13T11:59:06.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f2944640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5ec0a40f0 0x7fc5ec138080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:06.938 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f2944640 10 monclient(hunting): picked mon.noname-b con 0x7fc5ec0a40f0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:06.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f2944640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5ec0a45a0 con 0x7fc5ec06eeb0 2024-09-13T11:59:06.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f2944640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:06.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f2944640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7fc5ec09fde0 con 0x7fc5ec0a37d0 2024-09-13T11:59:06.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f2944640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:06.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f2944640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fc5ec06f330 con 0x7fc5ec0a40f0 2024-09-13T11:59:06.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f2944640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:06.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f2944640 10 monclient(hunting): authenticate will time out at 1127.522322s 2024-09-13T11:59:06.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f0ee1640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5ec0a40f0 0x7fc5ec138080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:06.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5ebfff640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc5ec0a37d0 0x7fc5ec0a3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:06.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f0ee1640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5ec0a40f0 0x7fc5ec138080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:51990/0 (socket says 172.21.15.100:51990) 2024-09-13T11:59:06.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f0ee1640 1 -- 172.21.15.100:0/381712556 learned_addr learned my addr 172.21.15.100:0/381712556 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:06.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f0ee1640 10 monclient(hunting): get_auth_request con 0x7fc5ec0a40f0 auth_method 0 2024-09-13T11:59:06.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f0ee1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:06.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f0ee1640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:06.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f0ee1640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:06.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5ebfff640 10 monclient(hunting): get_auth_request con 0x7fc5ec0a37d0 auth_method 0 2024-09-13T11:59:06.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:06.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:06.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:06.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f0ee1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:06.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f0ee1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:06.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f0ee1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:06.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:06.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:06.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:06.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f0ee1640 10 monclient(hunting): handle_auth_done global_id 24629 payload 274 2024-09-13T11:59:06.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f0ee1640 1 -- 172.21.15.100:0/381712556 >> v2:172.21.15.100:3300/0 conn(0x7fc5ec0a37d0 msgr2=0x7fc5ec0a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:06.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f0ee1640 1 --2- 172.21.15.100:0/381712556 >> v2:172.21.15.100:3300/0 conn(0x7fc5ec0a37d0 0x7fc5ec0a3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:06.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f0ee1640 1 -- 172.21.15.100:0/381712556 >> v1:172.21.15.100:6789/0 conn(0x7fc5ec06eeb0 legacy=0x7fc5ec137970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:06.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f0ee1640 10 monclient: _finish_hunting 0 2024-09-13T11:59:06.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f0ee1640 1 monclient: found mon.noname-b 2024-09-13T11:59:06.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f0ee1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:06.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f0ee1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:59:06.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5f0ee1640 1 -- 172.21.15.100:0/381712556 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5ec06e950 con 0x7fc5ec0a40f0 2024-09-13T11:59:06.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.928+0000 7fc5ebfff640 1 --2- 172.21.15.100:0/381712556 >> v2:172.21.15.100:3300/0 conn(0x7fc5ec0a37d0 0x7fc5ec0a3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:59:06.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.929+0000 7fc5f0ee1640 1 --2- 172.21.15.100:0/381712556 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5ec0a40f0 0x7fc5ec138080 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4009e50 tx=0x7fc5e402a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ec428d6e59a866b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:06.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.929+0000 7fc5eb7fe640 1 -- 172.21.15.100:0/381712556 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4036070 con 0x7fc5ec0a40f0 2024-09-13T11:59:06.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.929+0000 7fc5eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:06.942 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.929+0000 7fc5eb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:59:06.942 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.929+0000 7fc5eb7fe640 10 monclient: dump: 2024-09-13T11:59:06.942 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:06.942 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:06.942 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:06.942 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:06.942 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:06.942 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:06.942 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:06.942 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:06.942 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:06.942 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:06.942 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.929+0000 7fc5eb7fe640 10 monclient: _finish_auth 0 2024-09-13T11:59:06.943 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.929+0000 7fc5eb7fe640 10 monclient: _check_auth_tickets 2024-09-13T11:59:06.943 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.929+0000 7fc5eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:06.943 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.929+0000 7fc5eb7fe640 1 -- 172.21.15.100:0/381712556 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5e4031600 con 0x7fc5ec0a40f0 2024-09-13T11:59:06.943 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.929+0000 7fc5f2944640 5 monclient: authenticate success, global_id 24629 2024-09-13T11:59:06.943 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.929+0000 7fc5f2944640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:06.943 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.929+0000 7fc5eb7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:06.943 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.929+0000 7fc5f2944640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:06.943 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.929+0000 7fc5eb7fe640 1 -- 172.21.15.100:0/381712556 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e40319d0 con 0x7fc5ec0a40f0 2024-09-13T11:59:06.943 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.929+0000 7fc5eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:06.943 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.929+0000 7fc5f2944640 4 set_mon_vals no callback set 2024-09-13T11:59:06.943 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.929+0000 7fc5eb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:59:06.943 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.929+0000 7fc5eb7fe640 10 monclient: dump: 2024-09-13T11:59:06.943 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:06.944 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:06.944 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:06.944 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:06.944 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:06.944 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:06.944 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:06.944 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:06.944 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:06.944 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:06.944 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.930+0000 7fc5f2944640 10 monclient: shutdown 2024-09-13T11:59:06.944 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.930+0000 7fc5f2944640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:06.944 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.930+0000 7fc5f2944640 1 -- 172.21.15.100:0/381712556 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5ec0a40f0 msgr2=0x7fc5ec138080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:06.945 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.930+0000 7fc5f2944640 1 --2- 172.21.15.100:0/381712556 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5ec0a40f0 0x7fc5ec138080 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4009e50 tx=0x7fc5e402a5f0 comp rx=0 tx=0).stop 2024-09-13T11:59:06.945 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.930+0000 7fc5f2944640 1 -- 172.21.15.100:0/381712556 shutdown_connections 2024-09-13T11:59:06.945 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.930+0000 7fc5f2944640 1 --2- 172.21.15.100:0/381712556 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5ec0a40f0 0x7fc5ec138080 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:06.945 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.930+0000 7fc5f2944640 1 --2- 172.21.15.100:0/381712556 >> v2:172.21.15.100:3300/0 conn(0x7fc5ec0a37d0 0x7fc5ec0a3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:06.945 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.930+0000 7fc5f2944640 1 -- 172.21.15.100:0/381712556 >> 172.21.15.100:0/381712556 conn(0x7fc5ec0a83f0 msgr2=0x7fc5ec0a48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:06.945 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.930+0000 7fc5f2944640 1 -- 172.21.15.100:0/381712556 shutdown_connections 2024-09-13T11:59:06.945 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.930+0000 7fc5f2944640 1 -- 172.21.15.100:0/381712556 wait complete. 2024-09-13T11:59:06.945 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.930+0000 7fc5f2944640 10 monclient: build_initial_monmap 2024-09-13T11:59:06.945 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.930+0000 7fc5f2944640 10 monclient: monmap: 2024-09-13T11:59:06.945 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:06.945 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:06.945 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:06.945 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:06.946 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:06.946 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:06.946 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:06.946 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:06.946 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:06.946 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:06.946 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f2944640 1 Processor -- start 2024-09-13T11:59:06.946 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f2944640 1 -- start start 2024-09-13T11:59:06.946 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f2944640 10 monclient: init 2024-09-13T11:59:06.946 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f2944640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:06.946 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f2944640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:59:06.946 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f2944640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5ec06eeb0 0x7fc5ec1ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:06.946 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f2944640 10 monclient(hunting): picked mon.noname-a con 0x7fc5ec06eeb0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:06.947 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f2944640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5ec0a37d0 0x7fc5ec1cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:06.947 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f2944640 10 monclient(hunting): picked mon.noname-b con 0x7fc5ec0a37d0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:06.947 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f2944640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5ec0a40f0 0x7fc5ec1243f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:06.947 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f2944640 10 monclient(hunting): picked mon.noname-c con 0x7fc5ec0a40f0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:06.947 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f2944640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:06.947 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f2944640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ec09edd0 con 0x7fc5ec06eeb0 2024-09-13T11:59:06.947 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f2944640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:06.947 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f2944640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ec09ec50 con 0x7fc5ec0a37d0 2024-09-13T11:59:06.948 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f2944640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:06.948 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f2944640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fc5ec1385c0 con 0x7fc5ec0a40f0 2024-09-13T11:59:06.948 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f2944640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:06.948 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f2944640 10 monclient(hunting): authenticate will time out at 1127.525618s 2024-09-13T11:59:06.948 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5ebfff640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5ec06eeb0 0x7fc5ec1ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:06.948 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5ebfff640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5ec06eeb0 0x7fc5ec1ccc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47186/0 (socket says 172.21.15.100:47186) 2024-09-13T11:59:06.948 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5ebfff640 1 -- 172.21.15.100:0/2901377160 learned_addr learned my addr 172.21.15.100:0/2901377160 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:06.948 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5ebfff640 10 monclient(hunting): get_auth_request con 0x7fc5ec06eeb0 auth_method 0 2024-09-13T11:59:06.948 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:06.948 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:06.948 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:06.948 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f0ee1640 1 --2- 172.21.15.100:0/2901377160 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5ec0a40f0 0x7fc5ec1243f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:06.948 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f0ee1640 10 monclient(hunting): get_auth_request con 0x7fc5ec0a40f0 auth_method 0 2024-09-13T11:59:06.948 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f0ee1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:06.949 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f0ee1640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:06.949 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f0ee1640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:06.949 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:06.949 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:06.949 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:06.949 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f0ee1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:06.949 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f0ee1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:06.949 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5e3fff640 1 --2- 172.21.15.100:0/2901377160 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5ec0a37d0 0x7fc5ec1cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:06.949 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5f0ee1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:06.949 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5e3fff640 10 monclient(hunting): get_auth_request con 0x7fc5ec0a37d0 auth_method 0 2024-09-13T11:59:06.949 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:06.949 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:06.949 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:06.950 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5ebfff640 10 monclient(hunting): handle_auth_done global_id 15537 payload 824 2024-09-13T11:59:06.950 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5ebfff640 1 -- 172.21.15.100:0/2901377160 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5ec0a40f0 msgr2=0x7fc5ec1243f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:06.950 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5ebfff640 1 --2- 172.21.15.100:0/2901377160 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5ec0a40f0 0x7fc5ec1243f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:06.950 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5ebfff640 1 -- 172.21.15.100:0/2901377160 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5ec0a37d0 msgr2=0x7fc5ec1cd1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:06.950 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5ebfff640 1 --2- 172.21.15.100:0/2901377160 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5ec0a37d0 0x7fc5ec1cd1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:06.950 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5ebfff640 10 monclient: _finish_hunting 0 2024-09-13T11:59:06.950 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5ebfff640 1 monclient: found mon.noname-a 2024-09-13T11:59:06.950 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:06.950 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5ebfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:06.950 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.931+0000 7fc5ebfff640 1 -- 172.21.15.100:0/2901377160 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5ec09f0b0 con 0x7fc5ec06eeb0 2024-09-13T11:59:06.950 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.932+0000 7fc5e3fff640 1 --2- 172.21.15.100:0/2901377160 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5ec0a37d0 0x7fc5ec1cd1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:59:06.950 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.932+0000 7fc5ebfff640 1 --2- 172.21.15.100:0/2901377160 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5ec06eeb0 0x7fc5ec1ccc70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc00ebe0 tx=0x7fc5dc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:06.950 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.932+0000 7fc5e97fa640 1 -- 172.21.15.100:0/2901377160 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc017070 con 0x7fc5ec06eeb0 2024-09-13T11:59:06.951 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.932+0000 7fc5e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:06.951 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.932+0000 7fc5e97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:06.951 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.932+0000 7fc5e97fa640 10 monclient: dump: 2024-09-13T11:59:06.951 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:06.951 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:06.951 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:06.951 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:06.951 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:06.951 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:06.951 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:06.951 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:06.951 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:06.951 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:06.951 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.932+0000 7fc5e97fa640 10 monclient: _finish_auth 0 2024-09-13T11:59:06.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.932+0000 7fc5e97fa640 10 monclient: _check_auth_tickets 2024-09-13T11:59:06.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.932+0000 7fc5e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:06.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.932+0000 7fc5f2944640 5 monclient: authenticate success, global_id 15537 2024-09-13T11:59:06.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.932+0000 7fc5e97fa640 1 -- 172.21.15.100:0/2901377160 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5dc00cc30 con 0x7fc5ec06eeb0 2024-09-13T11:59:06.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.932+0000 7fc5e97fa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:06.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.932+0000 7fc5e97fa640 1 -- 172.21.15.100:0/2901377160 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc013470 con 0x7fc5ec06eeb0 2024-09-13T11:59:06.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.932+0000 7fc5f2944640 10 monclient: _renew_subs 2024-09-13T11:59:06.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.932+0000 7fc5f2944640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:06.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.932+0000 7fc5f2944640 1 -- 172.21.15.100:0/2901377160 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5ec125eb0 con 0x7fc5ec06eeb0 2024-09-13T11:59:06.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.932+0000 7fc5e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:06.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.932+0000 7fc5eaffd640 4 set_mon_vals no callback set 2024-09-13T11:59:06.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.932+0000 7fc5e97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:06.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.932+0000 7fc5e97fa640 10 monclient: dump: 2024-09-13T11:59:06.953 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:06.953 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:06.953 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:06.953 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:06.953 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:06.953 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:06.953 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:06.953 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:06.953 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:06.953 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:06.953 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.932+0000 7fc5f2944640 10 monclient: _renew_subs 2024-09-13T11:59:06.953 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.932+0000 7fc5f2944640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:06.953 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.932+0000 7fc5f2944640 1 -- 172.21.15.100:0/2901377160 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5ec126a10 con 0x7fc5ec06eeb0 2024-09-13T11:59:06.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.933+0000 7fc5e97fa640 1 -- 172.21.15.100:0/2901377160 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fc5dc01a070 con 0x7fc5ec06eeb0 2024-09-13T11:59:06.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.933+0000 7fc5f2944640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:06.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.933+0000 7fc5f2944640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:06.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.933+0000 7fc5f2944640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:06.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.933+0000 7fc5f2944640 1 -- 172.21.15.100:0/2901377160 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5ec0a3bb0 con 0x7fc5ec06eeb0 2024-09-13T11:59:06.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.933+0000 7fc5e97fa640 1 --2- 172.21.15.100:0/2901377160 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc5cc03dfa0 0x7fc5cc040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:06.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.933+0000 7fc5e97fa640 1 -- 172.21.15.100:0/2901377160 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7fc5dc052510 con 0x7fc5ec06eeb0 2024-09-13T11:59:06.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.933+0000 7fc5e3fff640 1 --2- 172.21.15.100:0/2901377160 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc5cc03dfa0 0x7fc5cc040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:06.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.933+0000 7fc5e3fff640 10 monclient: get_auth_request con 0x7fc5cc03dfa0 auth_method 0 2024-09-13T11:59:06.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.934+0000 7fc5e3fff640 1 --2- 172.21.15.100:0/2901377160 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc5cc03dfa0 0x7fc5cc040460 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4007fd0 tx=0x7fc5d4009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:06.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.937+0000 7fc5e97fa640 1 -- 172.21.15.100:0/2901377160 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc5dc023d00 con 0x7fc5ec06eeb0 2024-09-13T11:59:06.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.937+0000 7fc5e97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:06.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:06.937+0000 7fc5e97fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:59:07.113 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.111+0000 7fc5f2944640 1 -- 172.21.15.100:0/2901377160 --> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fc5ec128050 con 0x7fc5cc03dfa0 2024-09-13T11:59:07.114 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.112+0000 7fc5e97fa640 1 -- 172.21.15.100:0/2901377160 <== mgr.4114 v2:172.21.15.122:6824/441789830 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc5ec128050 con 0x7fc5cc03dfa0 2024-09-13T11:59:07.114 INFO:teuthology.orchestra.run.smithi100.stderr:instructed osd(s) 0 to deep-scrub 2024-09-13T11:59:07.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.114+0000 7fc5f2944640 1 -- 172.21.15.100:0/2901377160 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc5cc03dfa0 msgr2=0x7fc5cc040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.114+0000 7fc5f2944640 1 --2- 172.21.15.100:0/2901377160 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc5cc03dfa0 0x7fc5cc040460 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4007fd0 tx=0x7fc5d4009040 comp rx=0 tx=0).stop 2024-09-13T11:59:07.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.114+0000 7fc5f2944640 10 monclient: shutdown 2024-09-13T11:59:07.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.114+0000 7fc5f2944640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:07.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.114+0000 7fc5f2944640 1 -- 172.21.15.100:0/2901377160 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5ec06eeb0 msgr2=0x7fc5ec1ccc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.114+0000 7fc5f2944640 1 --2- 172.21.15.100:0/2901377160 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5ec06eeb0 0x7fc5ec1ccc70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc00ebe0 tx=0x7fc5dc00c520 comp rx=0 tx=0).stop 2024-09-13T11:59:07.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.114+0000 7fc5f2944640 1 -- 172.21.15.100:0/2901377160 shutdown_connections 2024-09-13T11:59:07.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.114+0000 7fc5f2944640 1 --2- 172.21.15.100:0/2901377160 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc5cc03dfa0 0x7fc5cc040460 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.114+0000 7fc5f2944640 1 --2- 172.21.15.100:0/2901377160 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5ec0a40f0 0x7fc5ec1243f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.114+0000 7fc5f2944640 1 --2- 172.21.15.100:0/2901377160 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5ec0a37d0 0x7fc5ec1cd1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.114+0000 7fc5f2944640 1 --2- 172.21.15.100:0/2901377160 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc5ec06eeb0 0x7fc5ec1ccc70 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.114+0000 7fc5f2944640 1 -- 172.21.15.100:0/2901377160 >> 172.21.15.100:0/2901377160 conn(0x7fc5ec0a83f0 msgr2=0x7fc5ec137e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:07.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.114+0000 7fc5f2944640 1 -- 172.21.15.100:0/2901377160 shutdown_connections 2024-09-13T11:59:07.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.114+0000 7fc5f2944640 1 -- 172.21.15.100:0/2901377160 wait complete. 2024-09-13T11:59:07.127 INFO:tasks.ceph:Scrubbing osd.1 2024-09-13T11:59:07.127 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:59:07.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.226+0000 7f2013496640 10 monclient: get_monmap_and_config 2024-09-13T11:59:07.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.226+0000 7f2013496640 10 monclient: build_initial_monmap 2024-09-13T11:59:07.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.226+0000 7f2013496640 10 monclient: monmap: 2024-09-13T11:59:07.229 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:07.229 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:07.229 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:07.228139+0000 2024-09-13T11:59:07.229 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:07.228139+0000 2024-09-13T11:59:07.229 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:07.229 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:07.229 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:07.229 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:07.229 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:07.229 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:07.229 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:07.230 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:07.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.226+0000 7f2013496640 1 Processor -- start 2024-09-13T11:59:07.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2013496640 1 -- start start 2024-09-13T11:59:07.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2013496640 10 monclient: init 2024-09-13T11:59:07.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2013496640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:07.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2013496640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-13T11:59:07.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2013496640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f200c0a5060 addr v1:172.21.15.122:6789/0 2024-09-13T11:59:07.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2013496640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f200c0a3f00 0x7f200c13cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2013496640 10 monclient(hunting): picked mon.noname-c con 0x7f200c0a3f00 addr v2:172.21.15.122:3300/0 2024-09-13T11:59:07.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2013496640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f200c0a4310 0x7f200c13f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2013496640 10 monclient(hunting): picked mon.noname-a con 0x7f200c0a4310 addr v2:172.21.15.100:3300/0 2024-09-13T11:59:07.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2013496640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f200c06ec40 con 0x7f200c0a5060 2024-09-13T11:59:07.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2013496640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:07.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2013496640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f200c0707b0 con 0x7f200c0a4310 2024-09-13T11:59:07.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2013496640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:07.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2013496640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f200c09eed0 con 0x7f200c0a3f00 2024-09-13T11:59:07.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2013496640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:07.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2013496640 10 monclient(hunting): authenticate will time out at 1127.821565s 2024-09-13T11:59:07.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2011a33640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f200c0a4310 0x7f200c13f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2011a33640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f200c0a4310 0x7f200c13f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47202/0 (socket says 172.21.15.100:47202) 2024-09-13T11:59:07.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2011a33640 1 -- 172.21.15.100:0/2929362446 learned_addr learned my addr 172.21.15.100:0/2929362446 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:07.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2011a33640 10 monclient(hunting): get_auth_request con 0x7f200c0a4310 auth_method 0 2024-09-13T11:59:07.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2011a33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:07.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2011a33640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:07.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2011a33640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:07.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2011a33640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:07.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2011a33640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:07.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2011a33640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:07.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2011a33640 10 monclient(hunting): handle_auth_done global_id 15540 payload 274 2024-09-13T11:59:07.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2010a31640 1 --2- 172.21.15.100:0/2929362446 >> v2:172.21.15.122:3300/0 conn(0x7f200c0a3f00 0x7f200c13cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2011a33640 1 -- 172.21.15.100:0/2929362446 >> v2:172.21.15.122:3300/0 conn(0x7f200c0a3f00 msgr2=0x7f200c13cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:07.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2011a33640 1 --2- 172.21.15.100:0/2929362446 >> v2:172.21.15.122:3300/0 conn(0x7f200c0a3f00 0x7f200c13cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2011a33640 1 -- 172.21.15.100:0/2929362446 >> v1:172.21.15.122:6789/0 conn(0x7f200c0a5060 legacy=0x7f200c0a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:07.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2011a33640 10 monclient: _finish_hunting 0 2024-09-13T11:59:07.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2011a33640 1 monclient: found mon.noname-a 2024-09-13T11:59:07.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2011a33640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:07.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2011a33640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:07.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.227+0000 7f2011a33640 1 -- 172.21.15.100:0/2929362446 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f200c06ee40 con 0x7f200c0a4310 2024-09-13T11:59:07.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.228+0000 7f2011a33640 1 --2- 172.21.15.100:0/2929362446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f200c0a4310 0x7f200c13f190 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f200c06eae0 tx=0x7f1ffc02a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa99f8ae1ac8bc5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:07.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.228+0000 7f200bfff640 1 -- 172.21.15.100:0/2929362446 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc036070 con 0x7f200c0a4310 2024-09-13T11:59:07.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.228+0000 7f200bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:07.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.228+0000 7f200bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:07.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.228+0000 7f200bfff640 10 monclient: dump: 2024-09-13T11:59:07.241 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:07.241 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:07.241 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.241 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.241 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:07.241 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:07.241 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:07.241 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:07.241 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:07.242 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:07.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.228+0000 7f200bfff640 10 monclient: _finish_auth 0 2024-09-13T11:59:07.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.228+0000 7f200bfff640 10 monclient: _check_auth_tickets 2024-09-13T11:59:07.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.228+0000 7f200bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:07.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.228+0000 7f200bfff640 1 -- 172.21.15.100:0/2929362446 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ffc031600 con 0x7f200c0a4310 2024-09-13T11:59:07.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.228+0000 7f2013496640 5 monclient: authenticate success, global_id 15540 2024-09-13T11:59:07.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.228+0000 7f2013496640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:07.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.228+0000 7f200bfff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:07.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.228+0000 7f200bfff640 1 -- 172.21.15.100:0/2929362446 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc0319d0 con 0x7f200c0a4310 2024-09-13T11:59:07.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.228+0000 7f2013496640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:07.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.228+0000 7f2013496640 4 set_mon_vals no callback set 2024-09-13T11:59:07.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.228+0000 7f200bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:07.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.228+0000 7f200bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:07.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.228+0000 7f200bfff640 10 monclient: dump: 2024-09-13T11:59:07.242 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:07.243 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:07.243 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.243 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.243 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:07.243 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:07.243 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:07.243 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:07.243 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:07.243 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:07.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.229+0000 7f2013496640 10 monclient: shutdown 2024-09-13T11:59:07.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.229+0000 7f2013496640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:07.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.229+0000 7f2013496640 1 -- 172.21.15.100:0/2929362446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f200c0a4310 msgr2=0x7f200c13f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.229+0000 7f2013496640 1 --2- 172.21.15.100:0/2929362446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f200c0a4310 0x7f200c13f190 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f200c06eae0 tx=0x7f1ffc02a560 comp rx=0 tx=0).stop 2024-09-13T11:59:07.244 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.229+0000 7f2013496640 1 -- 172.21.15.100:0/2929362446 shutdown_connections 2024-09-13T11:59:07.244 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.229+0000 7f2013496640 1 --2- 172.21.15.100:0/2929362446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f200c0a4310 0x7f200c13f190 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.244 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.229+0000 7f2013496640 1 --2- 172.21.15.100:0/2929362446 >> v2:172.21.15.122:3300/0 conn(0x7f200c0a3f00 0x7f200c13cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.244 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.229+0000 7f2013496640 1 -- 172.21.15.100:0/2929362446 >> 172.21.15.100:0/2929362446 conn(0x7f200c0a8480 msgr2=0x7f200c0a4950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:07.244 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.229+0000 7f2013496640 1 -- 172.21.15.100:0/2929362446 shutdown_connections 2024-09-13T11:59:07.244 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.229+0000 7f2013496640 1 -- 172.21.15.100:0/2929362446 wait complete. 2024-09-13T11:59:07.244 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.229+0000 7f2013496640 10 monclient: build_initial_monmap 2024-09-13T11:59:07.244 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.229+0000 7f2013496640 10 monclient: monmap: 2024-09-13T11:59:07.244 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:07.244 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:07.244 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:07.244 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:07.244 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:07.244 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:07.245 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:07.245 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:07.245 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:07.245 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:07.245 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2013496640 1 Processor -- start 2024-09-13T11:59:07.245 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2013496640 1 -- start start 2024-09-13T11:59:07.245 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2013496640 10 monclient: init 2024-09-13T11:59:07.245 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2013496640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:07.245 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2013496640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:59:07.245 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2013496640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f200c0a3f00 0x7f200c1cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.245 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2013496640 10 monclient(hunting): picked mon.noname-c con 0x7f200c0a3f00 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:07.245 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2013496640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f200c0a4310 0x7f200c137b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.245 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2013496640 10 monclient(hunting): picked mon.noname-b con 0x7f200c0a4310 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:07.246 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2013496640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f200c0a5060 0x7f200c13aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.246 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2013496640 10 monclient(hunting): picked mon.noname-a con 0x7f200c0a5060 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:07.246 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2013496640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:07.246 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2013496640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f200c06fc60 con 0x7f200c0a5060 2024-09-13T11:59:07.246 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2013496640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:07.246 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2013496640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f200c06f0c0 con 0x7f200c0a4310 2024-09-13T11:59:07.246 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2013496640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:07.246 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2013496640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f200c09ebc0 con 0x7f200c0a3f00 2024-09-13T11:59:07.246 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2013496640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:07.246 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2013496640 10 monclient(hunting): authenticate will time out at 1127.824720s 2024-09-13T11:59:07.246 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2011a33640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f200c0a5060 0x7f200c13aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.246 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2011232640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f200c0a3f00 0x7f200c1cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.247 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2011a33640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f200c0a5060 0x7f200c13aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47208/0 (socket says 172.21.15.100:47208) 2024-09-13T11:59:07.247 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2011232640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f200c0a3f00 0x7f200c1cc5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:52006/0 (socket says 172.21.15.100:52006) 2024-09-13T11:59:07.247 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2011a33640 1 -- 172.21.15.100:0/1125715395 learned_addr learned my addr 172.21.15.100:0/1125715395 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:07.247 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2011a33640 10 monclient(hunting): get_auth_request con 0x7f200c0a5060 auth_method 0 2024-09-13T11:59:07.247 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2011a33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:07.247 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2011a33640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:07.247 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2011a33640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:07.247 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2011232640 10 monclient(hunting): get_auth_request con 0x7f200c0a3f00 auth_method 0 2024-09-13T11:59:07.247 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2011232640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:07.247 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2011232640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:07.247 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2011232640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:07.247 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2010a31640 1 --2- 172.21.15.100:0/1125715395 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f200c0a4310 0x7f200c137b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.247 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2011a33640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:07.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2011a33640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:07.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2011a33640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:07.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2011232640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:07.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2011232640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:07.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2011232640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:07.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2010a31640 10 monclient(hunting): get_auth_request con 0x7f200c0a4310 auth_method 0 2024-09-13T11:59:07.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2010a31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:07.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2010a31640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:07.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.230+0000 7f2010a31640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:07.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2010a31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:07.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2010a31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:07.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2010a31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:07.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2011a33640 10 monclient(hunting): handle_auth_done global_id 15543 payload 824 2024-09-13T11:59:07.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2011a33640 1 -- 172.21.15.100:0/1125715395 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f200c0a3f00 msgr2=0x7f200c1cc5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2011a33640 1 --2- 172.21.15.100:0/1125715395 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f200c0a3f00 0x7f200c1cc5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2011a33640 1 -- 172.21.15.100:0/1125715395 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f200c0a4310 msgr2=0x7f200c137b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2011a33640 1 --2- 172.21.15.100:0/1125715395 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f200c0a4310 0x7f200c137b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2011a33640 10 monclient: _finish_hunting 0 2024-09-13T11:59:07.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2011a33640 1 monclient: found mon.noname-a 2024-09-13T11:59:07.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2011a33640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:07.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2011a33640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:07.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2011a33640 1 -- 172.21.15.100:0/1125715395 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f200c13f210 con 0x7f200c0a5060 2024-09-13T11:59:07.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2011232640 1 --2- 172.21.15.100:0/1125715395 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f200c0a3f00 0x7f200c1cc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:59:07.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2011a33640 1 --2- 172.21.15.100:0/1125715395 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f200c0a5060 0x7f200c13aa40 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc009f10 tx=0x7f1ffc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:07.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2009ffb640 1 -- 172.21.15.100:0/1125715395 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc036050 con 0x7f200c0a5060 2024-09-13T11:59:07.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2009ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:07.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2009ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:07.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2009ffb640 10 monclient: dump: 2024-09-13T11:59:07.250 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:07.250 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:07.250 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.250 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.250 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:07.250 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:07.250 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:07.250 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:07.250 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:07.250 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:07.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2009ffb640 10 monclient: _finish_auth 0 2024-09-13T11:59:07.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2009ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:59:07.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2009ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:07.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2009ffb640 1 -- 172.21.15.100:0/1125715395 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ffc042070 con 0x7f200c0a5060 2024-09-13T11:59:07.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2013496640 5 monclient: authenticate success, global_id 15543 2024-09-13T11:59:07.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2009ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:07.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2009ffb640 1 -- 172.21.15.100:0/1125715395 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc007bb0 con 0x7f200c0a5060 2024-09-13T11:59:07.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2013496640 10 monclient: _renew_subs 2024-09-13T11:59:07.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2013496640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:07.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2013496640 1 -- 172.21.15.100:0/1125715395 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f200c13c4b0 con 0x7f200c0a5060 2024-09-13T11:59:07.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2009ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:07.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2009ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:07.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2009ffb640 10 monclient: dump: 2024-09-13T11:59:07.251 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:07.251 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:07.251 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.251 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.252 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:07.252 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:07.252 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:07.252 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:07.252 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:07.252 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:07.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f200b7fe640 4 set_mon_vals no callback set 2024-09-13T11:59:07.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2013496640 10 monclient: _renew_subs 2024-09-13T11:59:07.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2013496640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:07.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.231+0000 7f2013496640 1 -- 172.21.15.100:0/1125715395 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f200c13e530 con 0x7f200c0a5060 2024-09-13T11:59:07.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.232+0000 7f2013496640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:07.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.232+0000 7f2013496640 1 -- 172.21.15.100:0/1125715395 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f200c09fd30 con 0x7f200c0a5060 2024-09-13T11:59:07.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.232+0000 7f2009ffb640 1 -- 172.21.15.100:0/1125715395 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f1ffc047460 con 0x7f200c0a5060 2024-09-13T11:59:07.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.232+0000 7f2009ffb640 1 --2- 172.21.15.100:0/1125715395 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f1ff803dfa0 0x7f1ff8040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.232+0000 7f2009ffb640 1 -- 172.21.15.100:0/1125715395 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7f1ffc073660 con 0x7f200c0a5060 2024-09-13T11:59:07.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.233+0000 7f2009ffb640 1 --2- 172.21.15.100:0/1125715395 >> [v2:172.21.15.100:6808/1385319817,v1:172.21.15.100:6809/1385319817] conn(0x7f1ff8043e10 0x7f1ff8046230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.233+0000 7f2009ffb640 1 -- 172.21.15.100:0/1125715395 --> [v2:172.21.15.100:6808/1385319817,v1:172.21.15.100:6809/1385319817] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1ffc031600 con 0x7f1ff8043e10 2024-09-13T11:59:07.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.233+0000 7f2009ffb640 1 -- 172.21.15.100:0/1125715395 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_get_version_reply(handle=1 version=313) ==== 24+0+0 (secure 0 0 0) 0x7f1ffc07a050 con 0x7f200c0a5060 2024-09-13T11:59:07.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.233+0000 7f2009ffb640 10 monclient: handle_get_version_reply finishing 1 version 313 2024-09-13T11:59:07.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.233+0000 7f2011232640 1 --2- 172.21.15.100:0/1125715395 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f1ff803dfa0 0x7f1ff8040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.233+0000 7f2011232640 10 monclient: get_auth_request con 0x7f1ff803dfa0 auth_method 0 2024-09-13T11:59:07.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.233+0000 7f2010a31640 1 --2- 172.21.15.100:0/1125715395 >> [v2:172.21.15.100:6808/1385319817,v1:172.21.15.100:6809/1385319817] conn(0x7f1ff8043e10 0x7f1ff8046230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.233+0000 7f2010a31640 10 monclient: get_auth_request con 0x7f1ff8043e10 auth_method 0 2024-09-13T11:59:07.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.233+0000 7f2010a31640 1 --2- 172.21.15.100:0/1125715395 >> [v2:172.21.15.100:6808/1385319817,v1:172.21.15.100:6809/1385319817] conn(0x7f1ff8043e10 0x7f1ff8046230 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:07.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.233+0000 7f2011232640 1 --2- 172.21.15.100:0/1125715395 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f1ff803dfa0 0x7f1ff8040460 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f200c13ec50 tx=0x7f2004009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:07.253 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.234+0000 7f2009ffb640 1 -- 172.21.15.100:0/1125715395 <== osd.1 v2:172.21.15.100:6808/1385319817 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1ffc031600 con 0x7f1ff8043e10 2024-09-13T11:59:07.254 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.246+0000 7f2013496640 1 -- 172.21.15.100:0/1125715395 --> [v2:172.21.15.100:6808/1385319817,v1:172.21.15.100:6809/1385319817] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f200c0a37f0 con 0x7f1ff8043e10 2024-09-13T11:59:07.254 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.246+0000 7f2009ffb640 1 -- 172.21.15.100:0/1125715395 <== osd.1 v2:172.21.15.100:6808/1385319817 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f200c0a37f0 con 0x7f1ff8043e10 2024-09-13T11:59:07.254 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:59:07.254 INFO:teuthology.orchestra.run.smithi100.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:59:07.255 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:59:07.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.246+0000 7f2013496640 1 -- 172.21.15.100:0/1125715395 >> [v2:172.21.15.100:6808/1385319817,v1:172.21.15.100:6809/1385319817] conn(0x7f1ff8043e10 msgr2=0x7f1ff8046230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.246+0000 7f2013496640 1 --2- 172.21.15.100:0/1125715395 >> [v2:172.21.15.100:6808/1385319817,v1:172.21.15.100:6809/1385319817] conn(0x7f1ff8043e10 0x7f1ff8046230 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.247+0000 7f2013496640 1 -- 172.21.15.100:0/1125715395 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f1ff803dfa0 msgr2=0x7f1ff8040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.247+0000 7f2013496640 1 --2- 172.21.15.100:0/1125715395 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f1ff803dfa0 0x7f1ff8040460 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f200c13ec50 tx=0x7f2004009040 comp rx=0 tx=0).stop 2024-09-13T11:59:07.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.247+0000 7f2013496640 10 monclient: shutdown 2024-09-13T11:59:07.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.247+0000 7f2013496640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:07.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.247+0000 7f2013496640 1 -- 172.21.15.100:0/1125715395 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f200c0a5060 msgr2=0x7f200c13aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.247+0000 7f2013496640 1 --2- 172.21.15.100:0/1125715395 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f200c0a5060 0x7f200c13aa40 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc009f10 tx=0x7f1ffc004a40 comp rx=0 tx=0).stop 2024-09-13T11:59:07.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.247+0000 7f2011a33640 1 -- 172.21.15.100:0/1125715395 reap_dead start 2024-09-13T11:59:07.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.247+0000 7f2013496640 1 -- 172.21.15.100:0/1125715395 shutdown_connections 2024-09-13T11:59:07.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.247+0000 7f2013496640 1 -- 172.21.15.100:0/1125715395 >> 172.21.15.100:0/1125715395 conn(0x7f200c0a8480 msgr2=0x7f200c0a0260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:07.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.247+0000 7f2013496640 1 -- 172.21.15.100:0/1125715395 shutdown_connections 2024-09-13T11:59:07.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.247+0000 7f2013496640 1 -- 172.21.15.100:0/1125715395 wait complete. 2024-09-13T11:59:07.258 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-13T11:59:07.356 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.354+0000 7f8d0c6c6640 10 monclient: get_monmap_and_config 2024-09-13T11:59:07.356 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.354+0000 7f8d0c6c6640 10 monclient: build_initial_monmap 2024-09-13T11:59:07.357 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.354+0000 7f8d0c6c6640 10 monclient: monmap: 2024-09-13T11:59:07.357 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:07.357 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:07.357 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:07.356426+0000 2024-09-13T11:59:07.357 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:07.356426+0000 2024-09-13T11:59:07.357 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:07.357 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:07.357 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:07.357 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:07.357 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:07.357 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:07.357 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:07.358 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:07.367 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.355+0000 7f8d0c6c6640 1 Processor -- start 2024-09-13T11:59:07.367 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.355+0000 7f8d0c6c6640 1 -- start start 2024-09-13T11:59:07.367 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.355+0000 7f8d0c6c6640 10 monclient: init 2024-09-13T11:59:07.367 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.355+0000 7f8d0c6c6640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:07.367 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.355+0000 7f8d0c6c6640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-13T11:59:07.367 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.355+0000 7f8d0c6c6640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8d040a1d80 0x7f8d040a2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.367 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.355+0000 7f8d0c6c6640 10 monclient(hunting): picked mon.noname-c con 0x7f8d040a1d80 addr v2:172.21.15.122:3300/0 2024-09-13T11:59:07.367 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.355+0000 7f8d0c6c6640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8d0406d470 0x7f8d040a26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.367 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.355+0000 7f8d0c6c6640 10 monclient(hunting): picked mon.noname-a con 0x7f8d0406d470 addr v2:172.21.15.100:3300/0 2024-09-13T11:59:07.367 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.355+0000 7f8d0c6c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8d04135620 addr v1:172.21.15.100:6789/0 2024-09-13T11:59:07.367 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.355+0000 7f8d0c6c6640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d0406d9a0 con 0x7f8d04135620 2024-09-13T11:59:07.367 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.355+0000 7f8d0c6c6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:07.368 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.355+0000 7f8d0c6c6640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f8d0406a330 con 0x7f8d0406d470 2024-09-13T11:59:07.368 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.355+0000 7f8d0c6c6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:07.368 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.355+0000 7f8d0c6c6640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f8d0409da10 con 0x7f8d040a1d80 2024-09-13T11:59:07.368 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.355+0000 7f8d0c6c6640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:07.368 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.355+0000 7f8d0c6c6640 10 monclient(hunting): authenticate will time out at 1127.949813s 2024-09-13T11:59:07.368 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.355+0000 7f8d09c61640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8d0406d470 0x7f8d040a26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.368 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.355+0000 7f8d09c61640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8d0406d470 0x7f8d040a26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47218/0 (socket says 172.21.15.100:47218) 2024-09-13T11:59:07.368 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.355+0000 7f8d09c61640 1 -- 172.21.15.100:0/3677766302 learned_addr learned my addr 172.21.15.100:0/3677766302 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:07.368 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.355+0000 7f8d09c61640 10 monclient(hunting): get_auth_request con 0x7f8d0406d470 auth_method 0 2024-09-13T11:59:07.368 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.355+0000 7f8d09c61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:07.368 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.355+0000 7f8d09c61640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:07.368 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.355+0000 7f8d09c61640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:07.369 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.356+0000 7f8d09c61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:07.369 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.356+0000 7f8d09c61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:07.369 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.356+0000 7f8d09c61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:07.369 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.356+0000 7f8d0a462640 1 --2- 172.21.15.100:0/3677766302 >> v2:172.21.15.122:3300/0 conn(0x7f8d040a1d80 0x7f8d040a2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.369 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.356+0000 7f8d0a462640 10 monclient(hunting): get_auth_request con 0x7f8d040a1d80 auth_method 0 2024-09-13T11:59:07.369 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.356+0000 7f8d0a462640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:07.369 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.356+0000 7f8d0a462640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:07.369 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.356+0000 7f8d0a462640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:07.369 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.356+0000 7f8d09c61640 10 monclient(hunting): handle_auth_done global_id 15546 payload 274 2024-09-13T11:59:07.369 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.356+0000 7f8d09c61640 1 -- 172.21.15.100:0/3677766302 >> v2:172.21.15.122:3300/0 conn(0x7f8d040a1d80 msgr2=0x7f8d040a2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:07.369 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.356+0000 7f8d09c61640 1 --2- 172.21.15.100:0/3677766302 >> v2:172.21.15.122:3300/0 conn(0x7f8d040a1d80 0x7f8d040a2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.369 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.356+0000 7f8d09c61640 1 -- 172.21.15.100:0/3677766302 >> v1:172.21.15.100:6789/0 conn(0x7f8d04135620 legacy=0x7f8d04135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.369 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.356+0000 7f8d09c61640 10 monclient: _finish_hunting 0 2024-09-13T11:59:07.370 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.356+0000 7f8d09c61640 1 monclient: found mon.noname-a 2024-09-13T11:59:07.370 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.356+0000 7f8d09c61640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:07.370 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.356+0000 7f8d09460640 1 -- 172.21.15.100:0/3677766302 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3322707581 0 0) 0x7f8d0406d9a0 con 0x7f8d04135620 2024-09-13T11:59:07.370 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.356+0000 7f8d09c61640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:07.370 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.356+0000 7f8d09c61640 1 -- 172.21.15.100:0/3677766302 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d040a3cc0 con 0x7f8d0406d470 2024-09-13T11:59:07.370 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.356+0000 7f8d0a462640 1 --2- 172.21.15.100:0/3677766302 >> v2:172.21.15.122:3300/0 conn(0x7f8d040a1d80 0x7f8d040a2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:59:07.370 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.356+0000 7f8d09460640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:59:07.370 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.356+0000 7f8d09c61640 1 --2- 172.21.15.100:0/3677766302 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d0406d470 0x7f8d040a26a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8d0406ef30 tx=0x7f8cf402a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c7c9f6481d5300 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:07.370 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d09460640 1 -- 172.21.15.100:0/3677766302 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4037070 con 0x7f8d0406d470 2024-09-13T11:59:07.370 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d09460640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:07.370 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d09460640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:07.370 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d09460640 10 monclient: dump: 2024-09-13T11:59:07.370 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:07.371 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:07.371 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.371 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.371 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:07.371 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:07.371 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:07.371 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:07.371 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:07.371 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:07.371 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d09460640 10 monclient: _finish_auth 0 2024-09-13T11:59:07.371 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d09460640 10 monclient: _check_auth_tickets 2024-09-13T11:59:07.371 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d09460640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:07.371 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d09460640 1 -- 172.21.15.100:0/3677766302 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cf4032440 con 0x7f8d0406d470 2024-09-13T11:59:07.372 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d0c6c6640 5 monclient: authenticate success, global_id 15546 2024-09-13T11:59:07.372 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d0c6c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:07.372 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d09460640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:07.372 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d0c6c6640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:07.372 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d09460640 1 -- 172.21.15.100:0/3677766302 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4032810 con 0x7f8d0406d470 2024-09-13T11:59:07.372 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d0c6c6640 4 set_mon_vals no callback set 2024-09-13T11:59:07.372 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d09460640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:07.372 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d09460640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:07.372 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d09460640 10 monclient: dump: 2024-09-13T11:59:07.372 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:07.372 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:07.372 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.373 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.373 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:07.373 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:07.373 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:07.373 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:07.373 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:07.373 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:07.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d0c6c6640 10 monclient: shutdown 2024-09-13T11:59:07.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d0c6c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:07.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d0c6c6640 1 -- 172.21.15.100:0/3677766302 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d0406d470 msgr2=0x7f8d040a26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d0c6c6640 1 --2- 172.21.15.100:0/3677766302 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d0406d470 0x7f8d040a26a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8d0406ef30 tx=0x7f8cf402a150 comp rx=0 tx=0).stop 2024-09-13T11:59:07.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d0c6c6640 1 -- 172.21.15.100:0/3677766302 shutdown_connections 2024-09-13T11:59:07.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d0c6c6640 1 --2- 172.21.15.100:0/3677766302 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d0406d470 0x7f8d040a26a0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.374 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d0c6c6640 1 --2- 172.21.15.100:0/3677766302 >> v2:172.21.15.122:3300/0 conn(0x7f8d040a1d80 0x7f8d040a2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.374 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d0c6c6640 1 -- 172.21.15.100:0/3677766302 >> 172.21.15.100:0/3677766302 conn(0x7f8d040a2e70 msgr2=0x7f8d040a3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:07.374 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d0c6c6640 1 -- 172.21.15.100:0/3677766302 shutdown_connections 2024-09-13T11:59:07.374 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.357+0000 7f8d0c6c6640 1 -- 172.21.15.100:0/3677766302 wait complete. 2024-09-13T11:59:07.374 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.358+0000 7f8d0c6c6640 10 monclient: build_initial_monmap 2024-09-13T11:59:07.374 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.358+0000 7f8d0c6c6640 10 monclient: monmap: 2024-09-13T11:59:07.374 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:07.374 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:07.374 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:07.374 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:07.374 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:07.374 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:07.374 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:07.375 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:07.375 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:07.375 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:07.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.358+0000 7f8d0c6c6640 1 Processor -- start 2024-09-13T11:59:07.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.358+0000 7f8d0c6c6640 1 -- start start 2024-09-13T11:59:07.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.358+0000 7f8d0c6c6640 10 monclient: init 2024-09-13T11:59:07.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0c6c6640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:07.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0c6c6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:59:07.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0c6c6640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d0406d470 0x7f8d041cc940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0c6c6640 10 monclient(hunting): picked mon.noname-a con 0x7f8d0406d470 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:07.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0c6c6640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d040a1d80 0x7f8d041cce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0c6c6640 10 monclient(hunting): picked mon.noname-c con 0x7f8d040a1d80 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:07.375 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0c6c6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d04135620 0x7f8d04131330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0c6c6640 10 monclient(hunting): picked mon.noname-b con 0x7f8d04135620 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:07.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0c6c6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:07.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0c6c6640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f8d040a2be0 con 0x7f8d0406d470 2024-09-13T11:59:07.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0c6c6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:07.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0c6c6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0409dcb0 con 0x7f8d04135620 2024-09-13T11:59:07.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0c6c6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:07.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0c6c6640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f8d0409c840 con 0x7f8d040a1d80 2024-09-13T11:59:07.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0c6c6640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:07.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0c6c6640 10 monclient(hunting): authenticate will time out at 1127.953110s 2024-09-13T11:59:07.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0a462640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d0406d470 0x7f8d041cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0a462640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d0406d470 0x7f8d041cc940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47220/0 (socket says 172.21.15.100:47220) 2024-09-13T11:59:07.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0a462640 1 -- 172.21.15.100:0/367010964 learned_addr learned my addr 172.21.15.100:0/367010964 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:07.376 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0a462640 10 monclient(hunting): get_auth_request con 0x7f8d0406d470 auth_method 0 2024-09-13T11:59:07.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0a462640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:07.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0a462640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:07.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0a462640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:07.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d09c61640 1 --2- 172.21.15.100:0/367010964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d040a1d80 0x7f8d041cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d09c61640 10 monclient(hunting): get_auth_request con 0x7f8d040a1d80 auth_method 0 2024-09-13T11:59:07.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d09c61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:07.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d09c61640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:07.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d09c61640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:07.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0a462640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:07.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0a462640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:07.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0a462640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:07.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d09c61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:07.377 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d09c61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:07.378 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d09c61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:07.378 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0a462640 10 monclient(hunting): handle_auth_done global_id 15552 payload 824 2024-09-13T11:59:07.378 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0ac63640 1 --2- 172.21.15.100:0/367010964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d04135620 0x7f8d04131330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.378 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0a462640 1 -- 172.21.15.100:0/367010964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d040a1d80 msgr2=0x7f8d041cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.378 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0a462640 1 --2- 172.21.15.100:0/367010964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d040a1d80 0x7f8d041cce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.378 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0a462640 1 -- 172.21.15.100:0/367010964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d04135620 msgr2=0x7f8d04131330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.378 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0a462640 1 --2- 172.21.15.100:0/367010964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d04135620 0x7f8d04131330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.378 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0a462640 10 monclient: _finish_hunting 0 2024-09-13T11:59:07.378 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0a462640 1 monclient: found mon.noname-a 2024-09-13T11:59:07.378 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0a462640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:07.378 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0a462640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:07.378 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0a462640 1 -- 172.21.15.100:0/367010964 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d040a5bc0 con 0x7f8d0406d470 2024-09-13T11:59:07.378 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d09c61640 1 --2- 172.21.15.100:0/367010964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d040a1d80 0x7f8d041cce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:59:07.379 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0a462640 1 --2- 172.21.15.100:0/367010964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d0406d470 0x7f8d041cc940 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f8d0000e730 tx=0x7f8d0000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:07.379 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8cfaffd640 1 -- 172.21.15.100:0/367010964 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0000f040 con 0x7f8d0406d470 2024-09-13T11:59:07.379 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8cfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:07.379 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8cfaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:07.379 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8cfaffd640 10 monclient: dump: 2024-09-13T11:59:07.379 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:07.379 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:07.379 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.379 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.379 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:07.379 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:07.379 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:07.379 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:07.379 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:07.380 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:07.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8cfaffd640 10 monclient: _finish_auth 0 2024-09-13T11:59:07.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8cfaffd640 10 monclient: _check_auth_tickets 2024-09-13T11:59:07.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8cfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:07.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8cfaffd640 1 -- 172.21.15.100:0/367010964 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d00004230 con 0x7f8d0406d470 2024-09-13T11:59:07.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0c6c6640 5 monclient: authenticate success, global_id 15552 2024-09-13T11:59:07.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8cfaffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:07.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8cfaffd640 1 -- 172.21.15.100:0/367010964 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0000ee40 con 0x7f8d0406d470 2024-09-13T11:59:07.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0c6c6640 10 monclient: _renew_subs 2024-09-13T11:59:07.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0c6c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:07.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0c6c6640 1 -- 172.21.15.100:0/367010964 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d040a5fa0 con 0x7f8d0406d470 2024-09-13T11:59:07.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8cfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:07.380 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8cfaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:07.381 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8cfaffd640 10 monclient: dump: 2024-09-13T11:59:07.381 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:07.381 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:07.381 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.381 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.381 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:07.381 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:07.381 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:07.381 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:07.381 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:07.381 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:07.381 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d08c5f640 4 set_mon_vals no callback set 2024-09-13T11:59:07.381 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0c6c6640 10 monclient: _renew_subs 2024-09-13T11:59:07.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0c6c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:07.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.359+0000 7f8d0c6c6640 1 -- 172.21.15.100:0/367010964 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d041346f0 con 0x7f8d0406d470 2024-09-13T11:59:07.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.360+0000 7f8d0c6c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:07.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.360+0000 7f8d0c6c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:07.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.360+0000 7f8d0c6c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:07.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.360+0000 7f8d0c6c6640 1 -- 172.21.15.100:0/367010964 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d041361f0 con 0x7f8d0406d470 2024-09-13T11:59:07.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.360+0000 7f8cfaffd640 1 -- 172.21.15.100:0/367010964 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f8d0000c6d0 con 0x7f8d0406d470 2024-09-13T11:59:07.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.360+0000 7f8cfaffd640 1 --2- 172.21.15.100:0/367010964 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f8ce803dfd0 0x7f8ce8040490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.360+0000 7f8cfaffd640 1 -- 172.21.15.100:0/367010964 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7f8d000523b0 con 0x7f8d0406d470 2024-09-13T11:59:07.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.361+0000 7f8d09c61640 1 --2- 172.21.15.100:0/367010964 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f8ce803dfd0 0x7f8ce8040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.361+0000 7f8d09c61640 10 monclient: get_auth_request con 0x7f8ce803dfd0 auth_method 0 2024-09-13T11:59:07.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.361+0000 7f8d09c61640 1 --2- 172.21.15.100:0/367010964 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f8ce803dfd0 0x7f8ce8040490 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f8cf4034000 tx=0x7f8cf402e000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:07.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.365+0000 7f8cfaffd640 1 -- 172.21.15.100:0/367010964 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f8d00017020 con 0x7f8d0406d470 2024-09-13T11:59:07.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.365+0000 7f8cfaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:07.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.365+0000 7f8cfaffd640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:59:07.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.538+0000 7f8d0c6c6640 1 -- 172.21.15.100:0/367010964 --> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f8d04069fa0 con 0x7f8ce803dfd0 2024-09-13T11:59:07.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.538+0000 7f8cfaffd640 1 -- 172.21.15.100:0/367010964 <== mgr.4114 v2:172.21.15.122:6824/441789830 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f8d04069fa0 con 0x7f8ce803dfd0 2024-09-13T11:59:07.541 INFO:teuthology.orchestra.run.smithi100.stderr:instructed osd(s) 1 to deep-scrub 2024-09-13T11:59:07.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.540+0000 7f8d0c6c6640 1 -- 172.21.15.100:0/367010964 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f8ce803dfd0 msgr2=0x7f8ce8040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.540+0000 7f8d0c6c6640 1 --2- 172.21.15.100:0/367010964 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f8ce803dfd0 0x7f8ce8040490 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f8cf4034000 tx=0x7f8cf402e000 comp rx=0 tx=0).stop 2024-09-13T11:59:07.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.540+0000 7f8d0c6c6640 10 monclient: shutdown 2024-09-13T11:59:07.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.540+0000 7f8d0c6c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:07.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.540+0000 7f8d0c6c6640 1 -- 172.21.15.100:0/367010964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d0406d470 msgr2=0x7f8d041cc940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.540+0000 7f8d0c6c6640 1 --2- 172.21.15.100:0/367010964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d0406d470 0x7f8d041cc940 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f8d0000e730 tx=0x7f8d0000ebf0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.541+0000 7f8d0c6c6640 1 -- 172.21.15.100:0/367010964 shutdown_connections 2024-09-13T11:59:07.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.541+0000 7f8d0c6c6640 1 --2- 172.21.15.100:0/367010964 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f8ce803dfd0 0x7f8ce8040490 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.541+0000 7f8d0c6c6640 1 --2- 172.21.15.100:0/367010964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d04135620 0x7f8d04131330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.541+0000 7f8d0c6c6640 1 --2- 172.21.15.100:0/367010964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d040a1d80 0x7f8d041cce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.541+0000 7f8d0c6c6640 1 --2- 172.21.15.100:0/367010964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d0406d470 0x7f8d041cc940 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.541+0000 7f8d0c6c6640 1 -- 172.21.15.100:0/367010964 >> 172.21.15.100:0/367010964 conn(0x7f8d040a2e70 msgr2=0x7f8d0412b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:07.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.541+0000 7f8d0c6c6640 1 -- 172.21.15.100:0/367010964 shutdown_connections 2024-09-13T11:59:07.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.541+0000 7f8d0c6c6640 1 -- 172.21.15.100:0/367010964 wait complete. 2024-09-13T11:59:07.553 INFO:tasks.ceph:Scrubbing osd.2 2024-09-13T11:59:07.553 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:59:07.655 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.652+0000 7ff65d144640 10 monclient: get_monmap_and_config 2024-09-13T11:59:07.655 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.652+0000 7ff65d144640 10 monclient: build_initial_monmap 2024-09-13T11:59:07.655 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.652+0000 7ff65d144640 10 monclient: monmap: 2024-09-13T11:59:07.656 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:07.656 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:07.656 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:07.654762+0000 2024-09-13T11:59:07.656 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:07.654762+0000 2024-09-13T11:59:07.656 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:07.656 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:07.656 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:07.656 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:07.656 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:07.656 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:07.657 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:07.657 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:07.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.653+0000 7ff65d144640 1 Processor -- start 2024-09-13T11:59:07.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.653+0000 7ff65d144640 1 -- start start 2024-09-13T11:59:07.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.653+0000 7ff65d144640 10 monclient: init 2024-09-13T11:59:07.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.653+0000 7ff65d144640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:07.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.653+0000 7ff65d144640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-13T11:59:07.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.653+0000 7ff65d144640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff6580a37f0 0x7ff6580a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.653+0000 7ff65d144640 10 monclient(hunting): picked mon.noname-c con 0x7ff6580a37f0 addr v2:172.21.15.122:3300/0 2024-09-13T11:59:07.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65d144640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff6580a4110 addr v1:172.21.15.100:6789/0 2024-09-13T11:59:07.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65d144640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff658138110 0x7ff65813fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65d144640 10 monclient(hunting): picked mon.noname-b con 0x7ff658138110 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:07.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65d144640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff65806eeb0 con 0x7ff6580a4110 2024-09-13T11:59:07.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65d144640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:07.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65d144640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7ff658070820 con 0x7ff6580a37f0 2024-09-13T11:59:07.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65d144640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:07.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65d144640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7ff6580a2320 con 0x7ff658138110 2024-09-13T11:59:07.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65d144640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:07.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65d144640 10 monclient(hunting): authenticate will time out at 1128.248048s 2024-09-13T11:59:07.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65659c640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7ff6580a4110 0x7ff658137a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:57124/0 (socket says 172.21.15.100:57124) 2024-09-13T11:59:07.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65759e640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff658138110 0x7ff65813fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65659c640 1 -- 172.21.15.100:0/2953825585 learned_addr learned my addr 172.21.15.100:0/2953825585 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:59:07.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65759e640 10 monclient(hunting): get_auth_request con 0x7ff658138110 auth_method 0 2024-09-13T11:59:07.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:07.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65759e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:07.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:07.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:07.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:07.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff656d9d640 1 --2- 172.21.15.100:0/2953825585 >> v2:172.21.15.122:3300/0 conn(0x7ff6580a37f0 0x7ff6580a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:07.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff656d9d640 10 monclient(hunting): get_auth_request con 0x7ff6580a37f0 auth_method 0 2024-09-13T11:59:07.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff656d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:07.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff656d9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:07.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff656d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:07.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff656d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:07.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff656d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:07.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff656d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:07.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff655d9b640 1 -- 172.21.15.100:0/2953825585 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 591228994 0 0) 0x7ff65806eeb0 con 0x7ff6580a4110 2024-09-13T11:59:07.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff655d9b640 10 monclient(hunting): _negotiate 2024-09-13T11:59:07.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff655d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:07.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff655d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:07.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff655d9b640 10 monclient(hunting): my global_id is 15555 2024-09-13T11:59:07.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff655d9b640 1 -- 172.21.15.100:0/2953825585 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff644003520 con 0x7ff6580a4110 2024-09-13T11:59:07.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65759e640 10 monclient(hunting): handle_auth_done global_id 24641 payload 274 2024-09-13T11:59:07.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65759e640 1 -- 172.21.15.100:0/2953825585 >> v2:172.21.15.122:3300/0 conn(0x7ff6580a37f0 msgr2=0x7ff6580a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:07.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65759e640 1 --2- 172.21.15.100:0/2953825585 >> v2:172.21.15.122:3300/0 conn(0x7ff6580a37f0 0x7ff6580a3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65759e640 1 -- 172.21.15.100:0/2953825585 >> v1:172.21.15.100:6789/0 conn(0x7ff6580a4110 legacy=0x7ff658137a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65759e640 10 monclient: _finish_hunting 0 2024-09-13T11:59:07.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65759e640 1 monclient: found mon.noname-b 2024-09-13T11:59:07.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:07.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65759e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:59:07.671 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.654+0000 7ff65759e640 1 -- 172.21.15.100:0/2953825585 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff658074330 con 0x7ff658138110 2024-09-13T11:59:07.671 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff65759e640 1 --2- 172.21.15.100:0/2953825585 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff658138110 0x7ff65813fc60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff6480097a0 tx=0x7ff64802a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99b4d715785895ec server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:07.671 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff655d9b640 1 -- 172.21.15.100:0/2953825585 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff648037070 con 0x7ff658138110 2024-09-13T11:59:07.671 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff655d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:07.671 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff655d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:59:07.671 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff655d9b640 10 monclient: dump: 2024-09-13T11:59:07.671 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:07.671 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:07.671 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.671 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.672 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:07.672 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:07.672 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:07.672 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:07.672 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:07.672 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:07.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff655d9b640 10 monclient: _finish_auth 0 2024-09-13T11:59:07.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff655d9b640 10 monclient: _check_auth_tickets 2024-09-13T11:59:07.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff655d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:07.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff655d9b640 1 -- 172.21.15.100:0/2953825585 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff64802adb0 con 0x7ff658138110 2024-09-13T11:59:07.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff65d144640 5 monclient: authenticate success, global_id 24641 2024-09-13T11:59:07.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff65d144640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:07.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff655d9b640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:07.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff65d144640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:07.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff655d9b640 1 -- 172.21.15.100:0/2953825585 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff648032660 con 0x7ff658138110 2024-09-13T11:59:07.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff65d144640 4 set_mon_vals no callback set 2024-09-13T11:59:07.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff655d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:07.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff655d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:59:07.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff655d9b640 10 monclient: dump: 2024-09-13T11:59:07.673 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:07.673 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:07.674 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.674 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.674 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:07.674 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:07.674 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:07.674 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:07.674 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:07.674 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:07.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff65d144640 10 monclient: shutdown 2024-09-13T11:59:07.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff65d144640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:07.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff65d144640 1 -- 172.21.15.100:0/2953825585 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff658138110 msgr2=0x7ff65813fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff65d144640 1 --2- 172.21.15.100:0/2953825585 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff658138110 0x7ff65813fc60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff6480097a0 tx=0x7ff64802a060 comp rx=0 tx=0).stop 2024-09-13T11:59:07.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff65d144640 1 -- 172.21.15.100:0/2953825585 shutdown_connections 2024-09-13T11:59:07.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff65d144640 1 --2- 172.21.15.100:0/2953825585 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff658138110 0x7ff65813fc60 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff65d144640 1 --2- 172.21.15.100:0/2953825585 >> v2:172.21.15.122:3300/0 conn(0x7ff6580a37f0 0x7ff6580a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff65d144640 1 -- 172.21.15.100:0/2953825585 >> 172.21.15.100:0/2953825585 conn(0x7ff6580a8480 msgr2=0x7ff6580a4950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:07.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff65d144640 1 -- 172.21.15.100:0/2953825585 shutdown_connections 2024-09-13T11:59:07.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.655+0000 7ff65d144640 1 -- 172.21.15.100:0/2953825585 wait complete. 2024-09-13T11:59:07.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.656+0000 7ff65d144640 10 monclient: build_initial_monmap 2024-09-13T11:59:07.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.656+0000 7ff65d144640 10 monclient: monmap: 2024-09-13T11:59:07.676 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:07.676 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:07.676 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:07.676 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:07.676 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:07.676 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:07.676 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:07.676 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:07.676 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:07.677 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:07.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65d144640 1 Processor -- start 2024-09-13T11:59:07.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65d144640 1 -- start start 2024-09-13T11:59:07.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65d144640 10 monclient: init 2024-09-13T11:59:07.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65d144640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:07.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65d144640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:59:07.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65d144640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6580a37f0 0x7ff658123f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65d144640 10 monclient(hunting): picked mon.noname-c con 0x7ff6580a37f0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:07.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65d144640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6580a4110 0x7ff658124470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65d144640 10 monclient(hunting): picked mon.noname-b con 0x7ff6580a4110 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:07.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65d144640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff658138110 0x7ff6581249b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65d144640 10 monclient(hunting): picked mon.noname-a con 0x7ff658138110 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:07.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65d144640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:07.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65d144640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7ff6580701d0 con 0x7ff658138110 2024-09-13T11:59:07.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65d144640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:07.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65d144640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff65806ea70 con 0x7ff6580a4110 2024-09-13T11:59:07.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65d144640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:07.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65d144640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7ff6580a4540 con 0x7ff6580a37f0 2024-09-13T11:59:07.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65d144640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:07.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65d144640 10 monclient(hunting): authenticate will time out at 1128.251732s 2024-09-13T11:59:07.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff656d9d640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6580a37f0 0x7ff658123f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff656d9d640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6580a37f0 0x7ff658123f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:52040/0 (socket says 172.21.15.100:52040) 2024-09-13T11:59:07.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff656d9d640 1 -- 172.21.15.100:0/1181543285 learned_addr learned my addr 172.21.15.100:0/1181543285 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:07.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff656d9d640 10 monclient(hunting): get_auth_request con 0x7ff6580a37f0 auth_method 0 2024-09-13T11:59:07.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff656d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:07.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff656d9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:07.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff656d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:07.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65759e640 1 --2- 172.21.15.100:0/1181543285 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff658138110 0x7ff6581249b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65659c640 1 --2- 172.21.15.100:0/1181543285 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6580a4110 0x7ff658124470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65759e640 10 monclient(hunting): get_auth_request con 0x7ff658138110 auth_method 0 2024-09-13T11:59:07.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:07.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65759e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:07.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:07.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff656d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:07.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff656d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:07.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff656d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:07.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65659c640 10 monclient(hunting): get_auth_request con 0x7ff6580a4110 auth_method 0 2024-09-13T11:59:07.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:07.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65659c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:07.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:07.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:07.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:07.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:07.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:07.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.657+0000 7ff65659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:07.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff65659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:07.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff656d9d640 10 monclient(hunting): handle_auth_done global_id 24644 payload 824 2024-09-13T11:59:07.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff656d9d640 1 -- 172.21.15.100:0/1181543285 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6580a4110 msgr2=0x7ff658124470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff656d9d640 1 --2- 172.21.15.100:0/1181543285 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6580a4110 0x7ff658124470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff656d9d640 1 -- 172.21.15.100:0/1181543285 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff658138110 msgr2=0x7ff6581249b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff656d9d640 1 --2- 172.21.15.100:0/1181543285 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff658138110 0x7ff6581249b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff656d9d640 10 monclient: _finish_hunting 0 2024-09-13T11:59:07.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff656d9d640 1 monclient: found mon.noname-c 2024-09-13T11:59:07.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff656d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:07.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff656d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:59:07.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff656d9d640 1 -- 172.21.15.100:0/1181543285 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff658126ee0 con 0x7ff6580a37f0 2024-09-13T11:59:07.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff656d9d640 1 --2- 172.21.15.100:0/1181543285 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6580a37f0 0x7ff658123f30 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff65000efc0 tx=0x7ff65000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:07.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff64f7fe640 1 -- 172.21.15.100:0/1181543285 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff650018020 con 0x7ff6580a37f0 2024-09-13T11:59:07.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff64f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:07.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff64f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:59:07.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff64f7fe640 10 monclient: dump: 2024-09-13T11:59:07.682 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:07.682 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:07.682 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.682 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.682 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:07.682 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:07.682 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:07.683 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:07.683 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:07.683 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:07.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff64f7fe640 10 monclient: _finish_auth 0 2024-09-13T11:59:07.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff64f7fe640 10 monclient: _check_auth_tickets 2024-09-13T11:59:07.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff64f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:07.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff65d144640 5 monclient: authenticate success, global_id 24644 2024-09-13T11:59:07.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff64f7fe640 1 -- 172.21.15.100:0/1181543285 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff650014480 con 0x7ff6580a37f0 2024-09-13T11:59:07.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff65d144640 10 monclient: _renew_subs 2024-09-13T11:59:07.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff65d144640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:59:07.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff65d144640 1 -- 172.21.15.100:0/1181543285 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff65809f0b0 con 0x7ff6580a37f0 2024-09-13T11:59:07.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff64f7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:07.683 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff64f7fe640 1 -- 172.21.15.100:0/1181543285 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff650014850 con 0x7ff6580a37f0 2024-09-13T11:59:07.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff64f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:07.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff64f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:59:07.684 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff64f7fe640 10 monclient: dump: 2024-09-13T11:59:07.684 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:07.684 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:07.684 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.684 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.684 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:07.684 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:07.684 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:07.684 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:07.684 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:07.685 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:07.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff65d144640 10 monclient: _renew_subs 2024-09-13T11:59:07.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff65559a640 4 set_mon_vals no callback set 2024-09-13T11:59:07.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff65d144640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:59:07.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.658+0000 7ff65d144640 1 -- 172.21.15.100:0/1181543285 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff658127a70 con 0x7ff6580a37f0 2024-09-13T11:59:07.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.659+0000 7ff65d144640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:59:07.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.659+0000 7ff65d144640 1 -- 172.21.15.100:0/1181543285 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff6580a53f0 con 0x7ff6580a37f0 2024-09-13T11:59:07.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.659+0000 7ff64f7fe640 1 -- 172.21.15.100:0/1181543285 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7ff650005d10 con 0x7ff6580a37f0 2024-09-13T11:59:07.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.659+0000 7ff64f7fe640 1 --2- 172.21.15.100:0/1181543285 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff63803dfa0 0x7ff638040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.685 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.659+0000 7ff64f7fe640 1 -- 172.21.15.100:0/1181543285 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7ff650053150 con 0x7ff6580a37f0 2024-09-13T11:59:07.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.659+0000 7ff64f7fe640 1 --2- 172.21.15.100:0/1181543285 >> [v2:172.21.15.100:6800/2354314235,v1:172.21.15.100:6801/2354314235] conn(0x7ff638043e10 0x7ff638046230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.659+0000 7ff64f7fe640 1 -- 172.21.15.100:0/1181543285 --> [v2:172.21.15.100:6800/2354314235,v1:172.21.15.100:6801/2354314235] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff650016e50 con 0x7ff638043e10 2024-09-13T11:59:07.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.659+0000 7ff64f7fe640 1 -- 172.21.15.100:0/1181543285 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_get_version_reply(handle=1 version=313) ==== 24+0+0 (secure 0 0 0) 0x7ff65005a050 con 0x7ff6580a37f0 2024-09-13T11:59:07.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.659+0000 7ff64f7fe640 10 monclient: handle_get_version_reply finishing 1 version 313 2024-09-13T11:59:07.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.659+0000 7ff65659c640 1 --2- 172.21.15.100:0/1181543285 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff63803dfa0 0x7ff638040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.659+0000 7ff65659c640 10 monclient: get_auth_request con 0x7ff63803dfa0 auth_method 0 2024-09-13T11:59:07.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.659+0000 7ff65759e640 1 --2- 172.21.15.100:0/1181543285 >> [v2:172.21.15.100:6800/2354314235,v1:172.21.15.100:6801/2354314235] conn(0x7ff638043e10 0x7ff638046230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.659+0000 7ff65759e640 10 monclient: get_auth_request con 0x7ff638043e10 auth_method 0 2024-09-13T11:59:07.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.660+0000 7ff65759e640 1 --2- 172.21.15.100:0/1181543285 >> [v2:172.21.15.100:6800/2354314235,v1:172.21.15.100:6801/2354314235] conn(0x7ff638043e10 0x7ff638046230 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:07.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.660+0000 7ff65659c640 1 --2- 172.21.15.100:0/1181543285 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff63803dfa0 0x7ff638040460 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff640007f50 tx=0x7ff640007ee0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:07.686 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.660+0000 7ff64f7fe640 1 -- 172.21.15.100:0/1181543285 <== osd.2 v2:172.21.15.100:6800/2354314235 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff650016e50 con 0x7ff638043e10 2024-09-13T11:59:07.687 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.672+0000 7ff65d144640 1 -- 172.21.15.100:0/1181543285 --> [v2:172.21.15.100:6800/2354314235,v1:172.21.15.100:6801/2354314235] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff658127e10 con 0x7ff638043e10 2024-09-13T11:59:07.687 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.673+0000 7ff64f7fe640 1 -- 172.21.15.100:0/1181543285 <== osd.2 v2:172.21.15.100:6800/2354314235 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff658127e10 con 0x7ff638043e10 2024-09-13T11:59:07.688 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:59:07.688 INFO:teuthology.orchestra.run.smithi100.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:59:07.688 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:59:07.689 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.673+0000 7ff65d144640 1 -- 172.21.15.100:0/1181543285 >> [v2:172.21.15.100:6800/2354314235,v1:172.21.15.100:6801/2354314235] conn(0x7ff638043e10 msgr2=0x7ff638046230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.673+0000 7ff65d144640 1 --2- 172.21.15.100:0/1181543285 >> [v2:172.21.15.100:6800/2354314235,v1:172.21.15.100:6801/2354314235] conn(0x7ff638043e10 0x7ff638046230 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.673+0000 7ff65d144640 1 -- 172.21.15.100:0/1181543285 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff63803dfa0 msgr2=0x7ff638040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.673+0000 7ff65d144640 1 --2- 172.21.15.100:0/1181543285 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7ff63803dfa0 0x7ff638040460 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff640007f50 tx=0x7ff640007ee0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.673+0000 7ff65d144640 10 monclient: shutdown 2024-09-13T11:59:07.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.673+0000 7ff65d144640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:07.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.673+0000 7ff65d144640 1 -- 172.21.15.100:0/1181543285 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6580a37f0 msgr2=0x7ff658123f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.673+0000 7ff65d144640 1 --2- 172.21.15.100:0/1181543285 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6580a37f0 0x7ff658123f30 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff65000efc0 tx=0x7ff65000c490 comp rx=0 tx=0).stop 2024-09-13T11:59:07.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.673+0000 7ff65759e640 1 -- 172.21.15.100:0/1181543285 reap_dead start 2024-09-13T11:59:07.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.673+0000 7ff65d144640 1 -- 172.21.15.100:0/1181543285 shutdown_connections 2024-09-13T11:59:07.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.673+0000 7ff65d144640 1 -- 172.21.15.100:0/1181543285 >> 172.21.15.100:0/1181543285 conn(0x7ff6580a8480 msgr2=0x7ff6580a6ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:07.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.673+0000 7ff65d144640 1 -- 172.21.15.100:0/1181543285 shutdown_connections 2024-09-13T11:59:07.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.673+0000 7ff65d144640 1 -- 172.21.15.100:0/1181543285 wait complete. 2024-09-13T11:59:07.691 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-13T11:59:07.789 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.787+0000 7fc13bfaa640 10 monclient: get_monmap_and_config 2024-09-13T11:59:07.789 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.787+0000 7fc13bfaa640 10 monclient: build_initial_monmap 2024-09-13T11:59:07.790 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.787+0000 7fc13bfaa640 10 monclient: monmap: 2024-09-13T11:59:07.790 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:07.790 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:07.790 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:07.789409+0000 2024-09-13T11:59:07.790 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:07.789409+0000 2024-09-13T11:59:07.790 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:07.790 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:07.790 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:07.790 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:07.790 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:07.790 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:07.790 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:07.790 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:07.799 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.788+0000 7fc13bfaa640 1 Processor -- start 2024-09-13T11:59:07.799 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.788+0000 7fc13bfaa640 1 -- start start 2024-09-13T11:59:07.799 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.788+0000 7fc13bfaa640 10 monclient: init 2024-09-13T11:59:07.799 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.788+0000 7fc13bfaa640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:07.799 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.788+0000 7fc13bfaa640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-13T11:59:07.799 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.788+0000 7fc13bfaa640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc1340a37d0 0x7fc1340a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.799 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.788+0000 7fc13bfaa640 10 monclient(hunting): picked mon.noname-c con 0x7fc1340a37d0 addr v2:172.21.15.122:3300/0 2024-09-13T11:59:07.800 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.788+0000 7fc13bfaa640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc13409fd90 0x7fc1340a40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.800 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.788+0000 7fc13bfaa640 10 monclient(hunting): picked mon.noname-b con 0x7fc13409fd90 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:07.800 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.788+0000 7fc13bfaa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc13413b990 addr v1:172.21.15.122:6789/0 2024-09-13T11:59:07.800 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.788+0000 7fc13bfaa640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc13406fe10 con 0x7fc13413b990 2024-09-13T11:59:07.800 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.788+0000 7fc13bfaa640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:07.800 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.788+0000 7fc13bfaa640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7fc1340a4630 con 0x7fc1340a37d0 2024-09-13T11:59:07.800 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.788+0000 7fc13bfaa640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:07.800 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.788+0000 7fc13bfaa640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fc13406eff0 con 0x7fc13409fd90 2024-09-13T11:59:07.800 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.788+0000 7fc13bfaa640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:07.800 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.788+0000 7fc13bfaa640 10 monclient(hunting): authenticate will time out at 1128.382937s 2024-09-13T11:59:07.800 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139545640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc13409fd90 0x7fc1340a40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.800 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139545640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc13409fd90 0x7fc1340a40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:52054/0 (socket says 172.21.15.100:52054) 2024-09-13T11:59:07.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139545640 1 -- 172.21.15.100:0/973696735 learned_addr learned my addr 172.21.15.100:0/973696735 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:07.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139545640 10 monclient(hunting): get_auth_request con 0x7fc13409fd90 auth_method 0 2024-09-13T11:59:07.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139545640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:07.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139545640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:07.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139545640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:07.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139545640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:07.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139545640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:07.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139d46640 1 --2- 172.21.15.100:0/973696735 >> v2:172.21.15.122:3300/0 conn(0x7fc1340a37d0 0x7fc1340a3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139545640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:07.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139d46640 10 monclient(hunting): get_auth_request con 0x7fc1340a37d0 auth_method 0 2024-09-13T11:59:07.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139d46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:07.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139d46640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:07.801 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139d46640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:07.802 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139545640 10 monclient(hunting): handle_auth_done global_id 24647 payload 274 2024-09-13T11:59:07.802 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139545640 1 -- 172.21.15.100:0/973696735 >> v2:172.21.15.122:3300/0 conn(0x7fc1340a37d0 msgr2=0x7fc1340a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:07.802 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139545640 1 --2- 172.21.15.100:0/973696735 >> v2:172.21.15.122:3300/0 conn(0x7fc1340a37d0 0x7fc1340a3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.802 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139545640 1 -- 172.21.15.100:0/973696735 >> v1:172.21.15.122:6789/0 conn(0x7fc13413b990 legacy=0x7fc13413dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.802 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139545640 10 monclient: _finish_hunting 0 2024-09-13T11:59:07.802 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139545640 1 monclient: found mon.noname-b 2024-09-13T11:59:07.802 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139545640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:07.802 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139545640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:59:07.802 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139545640 1 -- 172.21.15.100:0/973696735 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc134070310 con 0x7fc13409fd90 2024-09-13T11:59:07.802 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139d46640 1 --2- 172.21.15.100:0/973696735 >> v2:172.21.15.122:3300/0 conn(0x7fc1340a37d0 0x7fc1340a3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:59:07.802 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.789+0000 7fc139545640 1 --2- 172.21.15.100:0/973696735 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc13409fd90 0x7fc1340a40f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc124009870 tx=0x7fc12402a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=409403c768b9355c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:07.802 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc138d44640 1 -- 172.21.15.100:0/973696735 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc124037070 con 0x7fc13409fd90 2024-09-13T11:59:07.802 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc138d44640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:07.802 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc138d44640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:59:07.803 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc138d44640 10 monclient: dump: 2024-09-13T11:59:07.803 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:07.803 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:07.803 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.803 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.803 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:07.803 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:07.803 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:07.803 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:07.803 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:07.803 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:07.803 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc138d44640 10 monclient: _finish_auth 0 2024-09-13T11:59:07.803 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc138d44640 10 monclient: _check_auth_tickets 2024-09-13T11:59:07.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc138d44640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:07.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc138d44640 1 -- 172.21.15.100:0/973696735 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc124032440 con 0x7fc13409fd90 2024-09-13T11:59:07.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc13bfaa640 5 monclient: authenticate success, global_id 24647 2024-09-13T11:59:07.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc13bfaa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:07.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc138d44640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:07.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc13bfaa640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:07.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc138d44640 1 -- 172.21.15.100:0/973696735 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc124032810 con 0x7fc13409fd90 2024-09-13T11:59:07.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc138d44640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:07.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc13bfaa640 4 set_mon_vals no callback set 2024-09-13T11:59:07.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc138d44640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:59:07.804 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc138d44640 10 monclient: dump: 2024-09-13T11:59:07.804 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:07.804 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:07.804 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.805 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.805 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:07.805 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:07.805 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:07.805 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:07.805 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:07.805 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:07.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc13bfaa640 10 monclient: shutdown 2024-09-13T11:59:07.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc13bfaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:07.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc13bfaa640 1 -- 172.21.15.100:0/973696735 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc13409fd90 msgr2=0x7fc1340a40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc13bfaa640 1 --2- 172.21.15.100:0/973696735 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc13409fd90 0x7fc1340a40f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc124009870 tx=0x7fc12402a130 comp rx=0 tx=0).stop 2024-09-13T11:59:07.805 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc13bfaa640 1 -- 172.21.15.100:0/973696735 shutdown_connections 2024-09-13T11:59:07.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc13bfaa640 1 --2- 172.21.15.100:0/973696735 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc13409fd90 0x7fc1340a40f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc13bfaa640 1 --2- 172.21.15.100:0/973696735 >> v2:172.21.15.122:3300/0 conn(0x7fc1340a37d0 0x7fc1340a3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc13bfaa640 1 -- 172.21.15.100:0/973696735 >> 172.21.15.100:0/973696735 conn(0x7fc1340a83f0 msgr2=0x7fc1340a48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:07.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc13bfaa640 1 -- 172.21.15.100:0/973696735 shutdown_connections 2024-09-13T11:59:07.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.790+0000 7fc13bfaa640 1 -- 172.21.15.100:0/973696735 wait complete. 2024-09-13T11:59:07.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.791+0000 7fc13bfaa640 10 monclient: build_initial_monmap 2024-09-13T11:59:07.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.791+0000 7fc13bfaa640 10 monclient: monmap: 2024-09-13T11:59:07.806 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:07.806 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:07.806 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:07.806 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:07.806 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:07.807 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:07.807 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:07.807 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:07.807 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:07.807 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:07.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.791+0000 7fc13bfaa640 1 Processor -- start 2024-09-13T11:59:07.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.791+0000 7fc13bfaa640 1 -- start start 2024-09-13T11:59:07.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.791+0000 7fc13bfaa640 10 monclient: init 2024-09-13T11:59:07.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13bfaa640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:07.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13bfaa640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:59:07.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13bfaa640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc13409fd90 0x7fc13409d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13bfaa640 10 monclient(hunting): picked mon.noname-c con 0x7fc13409fd90 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:07.807 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13bfaa640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc1340a37d0 0x7fc13413aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13bfaa640 10 monclient(hunting): picked mon.noname-a con 0x7fc1340a37d0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13bfaa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc13413b990 0x7fc1341cc770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13bfaa640 10 monclient(hunting): picked mon.noname-b con 0x7fc13413b990 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13bfaa640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13bfaa640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fc1340a51b0 con 0x7fc1340a37d0 2024-09-13T11:59:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13bfaa640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13bfaa640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc1340a5330 con 0x7fc13413b990 2024-09-13T11:59:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13bfaa640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13bfaa640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fc13413f4a0 con 0x7fc13409fd90 2024-09-13T11:59:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13bfaa640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13bfaa640 10 monclient(hunting): authenticate will time out at 1128.386145s 2024-09-13T11:59:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139d46640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc13409fd90 0x7fc13409d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139d46640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc13409fd90 0x7fc13409d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:52064/0 (socket says 172.21.15.100:52064) 2024-09-13T11:59:07.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139545640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc1340a37d0 0x7fc13413aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139d46640 1 -- 172.21.15.100:0/3465767310 learned_addr learned my addr 172.21.15.100:0/3465767310 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:07.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139d46640 10 monclient(hunting): get_auth_request con 0x7fc13409fd90 auth_method 0 2024-09-13T11:59:07.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139d46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:07.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139d46640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:07.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139d46640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:07.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139545640 10 monclient(hunting): get_auth_request con 0x7fc1340a37d0 auth_method 0 2024-09-13T11:59:07.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139545640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:07.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139545640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:07.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139545640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:07.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139d46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:07.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139d46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:07.809 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139d46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:07.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139545640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:07.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139545640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:07.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139545640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:07.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13a547640 1 --2- 172.21.15.100:0/3465767310 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc13413b990 0x7fc1341cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13a547640 10 monclient(hunting): get_auth_request con 0x7fc13413b990 auth_method 0 2024-09-13T11:59:07.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13a547640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:07.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13a547640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:07.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13a547640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:07.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13a547640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:07.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13a547640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:07.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13a547640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:07.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139545640 10 monclient(hunting): handle_auth_done global_id 15561 payload 824 2024-09-13T11:59:07.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139545640 1 -- 172.21.15.100:0/3465767310 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc13409fd90 msgr2=0x7fc13409d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139545640 1 --2- 172.21.15.100:0/3465767310 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc13409fd90 0x7fc13409d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139545640 1 -- 172.21.15.100:0/3465767310 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc13413b990 msgr2=0x7fc1341cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139545640 1 --2- 172.21.15.100:0/3465767310 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc13413b990 0x7fc1341cc770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139545640 10 monclient: _finish_hunting 0 2024-09-13T11:59:07.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139545640 1 monclient: found mon.noname-a 2024-09-13T11:59:07.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139545640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:07.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139545640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:07.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139545640 1 -- 172.21.15.100:0/3465767310 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc134125470 con 0x7fc1340a37d0 2024-09-13T11:59:07.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139d46640 1 --2- 172.21.15.100:0/3465767310 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc13409fd90 0x7fc13409d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:59:07.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc139545640 1 --2- 172.21.15.100:0/3465767310 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc1340a37d0 0x7fc13413aac0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fc124031000 tx=0x7fc1240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:07.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc12a7fc640 1 -- 172.21.15.100:0/3465767310 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc124041020 con 0x7fc1340a37d0 2024-09-13T11:59:07.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc12a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:07.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc12a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:07.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc12a7fc640 10 monclient: dump: 2024-09-13T11:59:07.812 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:07.812 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:07.812 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.812 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.812 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:07.812 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:07.812 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:07.812 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:07.812 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:07.812 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:07.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc12a7fc640 10 monclient: _finish_auth 0 2024-09-13T11:59:07.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc12a7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:59:07.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc12a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:07.812 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13bfaa640 5 monclient: authenticate success, global_id 15561 2024-09-13T11:59:07.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc12a7fc640 1 -- 172.21.15.100:0/3465767310 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc124030760 con 0x7fc1340a37d0 2024-09-13T11:59:07.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13bfaa640 10 monclient: _renew_subs 2024-09-13T11:59:07.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13bfaa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:07.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13bfaa640 1 -- 172.21.15.100:0/3465767310 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc13413b780 con 0x7fc1340a37d0 2024-09-13T11:59:07.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc12a7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:07.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc12a7fc640 1 -- 172.21.15.100:0/3465767310 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc124037040 con 0x7fc1340a37d0 2024-09-13T11:59:07.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc12a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:07.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc12a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:07.813 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc12a7fc640 10 monclient: dump: 2024-09-13T11:59:07.813 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:07.813 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:07.813 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.813 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:07.814 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:07.814 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:07.814 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:07.814 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:07.814 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:07.814 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:07.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13bfaa640 10 monclient: _renew_subs 2024-09-13T11:59:07.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc12bfff640 4 set_mon_vals no callback set 2024-09-13T11:59:07.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13bfaa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:07.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.792+0000 7fc13bfaa640 1 -- 172.21.15.100:0/3465767310 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc13409e3c0 con 0x7fc1340a37d0 2024-09-13T11:59:07.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.793+0000 7fc12a7fc640 1 -- 172.21.15.100:0/3465767310 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fc124044020 con 0x7fc1340a37d0 2024-09-13T11:59:07.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.793+0000 7fc13bfaa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:07.814 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.793+0000 7fc13bfaa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:07.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.793+0000 7fc13bfaa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:07.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.793+0000 7fc13bfaa640 1 -- 172.21.15.100:0/3465767310 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1340a3bb0 con 0x7fc1340a37d0 2024-09-13T11:59:07.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.794+0000 7fc12a7fc640 1 --2- 172.21.15.100:0/3465767310 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc11803dfa0 0x7fc118040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:07.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.794+0000 7fc12a7fc640 1 -- 172.21.15.100:0/3465767310 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7fc124072e00 con 0x7fc1340a37d0 2024-09-13T11:59:07.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.794+0000 7fc139d46640 1 --2- 172.21.15.100:0/3465767310 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc11803dfa0 0x7fc118040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:07.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.794+0000 7fc139d46640 10 monclient: get_auth_request con 0x7fc11803dfa0 auth_method 0 2024-09-13T11:59:07.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.794+0000 7fc139d46640 1 --2- 172.21.15.100:0/3465767310 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc11803dfa0 0x7fc118040460 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc130003c20 tx=0x7fc1300099d0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:07.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.798+0000 7fc12a7fc640 1 -- 172.21.15.100:0/3465767310 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc12404b030 con 0x7fc1340a37d0 2024-09-13T11:59:07.815 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.798+0000 7fc12a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:07.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.798+0000 7fc12a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:59:07.975 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.972+0000 7fc13bfaa640 1 -- 172.21.15.100:0/3465767310 --> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fc13406ca60 con 0x7fc11803dfa0 2024-09-13T11:59:07.975 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.973+0000 7fc12a7fc640 1 -- 172.21.15.100:0/3465767310 <== mgr.4114 v2:172.21.15.122:6824/441789830 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc13406ca60 con 0x7fc11803dfa0 2024-09-13T11:59:07.975 INFO:teuthology.orchestra.run.smithi100.stderr:instructed osd(s) 2 to deep-scrub 2024-09-13T11:59:07.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.975+0000 7fc13bfaa640 1 -- 172.21.15.100:0/3465767310 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc11803dfa0 msgr2=0x7fc118040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.975+0000 7fc13bfaa640 1 --2- 172.21.15.100:0/3465767310 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc11803dfa0 0x7fc118040460 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc130003c20 tx=0x7fc1300099d0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.975+0000 7fc13bfaa640 10 monclient: shutdown 2024-09-13T11:59:07.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.975+0000 7fc13bfaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:07.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.975+0000 7fc13bfaa640 1 -- 172.21.15.100:0/3465767310 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc1340a37d0 msgr2=0x7fc13413aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:07.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.975+0000 7fc13bfaa640 1 --2- 172.21.15.100:0/3465767310 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc1340a37d0 0x7fc13413aac0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fc124031000 tx=0x7fc1240046c0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.975+0000 7fc13bfaa640 1 -- 172.21.15.100:0/3465767310 shutdown_connections 2024-09-13T11:59:07.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.975+0000 7fc13bfaa640 1 --2- 172.21.15.100:0/3465767310 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc11803dfa0 0x7fc118040460 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.975+0000 7fc13bfaa640 1 --2- 172.21.15.100:0/3465767310 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc1340a37d0 0x7fc13413aac0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.975+0000 7fc13bfaa640 1 --2- 172.21.15.100:0/3465767310 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc13413b990 0x7fc1341cc770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.975+0000 7fc13bfaa640 1 --2- 172.21.15.100:0/3465767310 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc13409fd90 0x7fc13409d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:07.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.975+0000 7fc13bfaa640 1 -- 172.21.15.100:0/3465767310 >> 172.21.15.100:0/3465767310 conn(0x7fc1340a83f0 msgr2=0x7fc13413dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:07.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.975+0000 7fc13bfaa640 1 -- 172.21.15.100:0/3465767310 shutdown_connections 2024-09-13T11:59:07.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:07.975+0000 7fc13bfaa640 1 -- 172.21.15.100:0/3465767310 wait complete. 2024-09-13T11:59:07.987 INFO:tasks.ceph:Scrubbing osd.3 2024-09-13T11:59:07.988 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:59:08.089 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.086+0000 7f567350e640 10 monclient: get_monmap_and_config 2024-09-13T11:59:08.089 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.086+0000 7f567350e640 10 monclient: build_initial_monmap 2024-09-13T11:59:08.089 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.086+0000 7f567350e640 10 monclient: monmap: 2024-09-13T11:59:08.089 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:08.089 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.089 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:08.088815+0000 2024-09-13T11:59:08.089 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:08.088815+0000 2024-09-13T11:59:08.089 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:08.089 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.090 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:08.090 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:08.090 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:08.090 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:08.090 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:08.090 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.098 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.087+0000 7f567350e640 1 Processor -- start 2024-09-13T11:59:08.098 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.087+0000 7f567350e640 1 -- start start 2024-09-13T11:59:08.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.087+0000 7f567350e640 10 monclient: init 2024-09-13T11:59:08.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f567350e640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:08.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f567350e640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-13T11:59:08.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f567350e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f566c09cb10 0x7f566c09cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f567350e640 10 monclient(hunting): picked mon.noname-c con 0x7f566c09cb10 addr v2:172.21.15.122:3300/0 2024-09-13T11:59:08.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f567350e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f566c09f3a0 addr v1:172.21.15.100:6789/0 2024-09-13T11:59:08.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f567350e640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f566c139530 0x7f566c13b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f567350e640 10 monclient(hunting): picked mon.noname-b con 0x7f566c139530 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:08.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f567350e640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f566c09e6e0 con 0x7f566c09f3a0 2024-09-13T11:59:08.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f567350e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:08.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f567350e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f566c09e9c0 con 0x7f566c09cb10 2024-09-13T11:59:08.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f567350e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:08.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f567350e640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f566c0a17a0 con 0x7f566c139530 2024-09-13T11:59:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f567350e640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f567350e640 10 monclient(hunting): authenticate will time out at 1128.682336s 2024-09-13T11:59:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f5670aa9640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f566c09f3a0 0x7f566c09d430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:57136/0 (socket says 172.21.15.100:57136) 2024-09-13T11:59:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f5671aab640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f566c139530 0x7f566c13b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f5670aa9640 1 -- 172.21.15.100:0/2498122087 learned_addr learned my addr 172.21.15.100:0/2498122087 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:59:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f5671aab640 10 monclient(hunting): get_auth_request con 0x7f566c139530 auth_method 0 2024-09-13T11:59:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f5671aab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f5671aab640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f5671aab640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f5671aab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f5671aab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f56712aa640 1 --2- 172.21.15.100:0/2498122087 >> v2:172.21.15.122:3300/0 conn(0x7f566c09cb10 0x7f566c09cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f5671aab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:08.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f56712aa640 10 monclient(hunting): get_auth_request con 0x7f566c09cb10 auth_method 0 2024-09-13T11:59:08.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f56712aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:08.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f56712aa640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f56712aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f56712aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:08.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f56712aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:08.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f56712aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:08.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f5671aab640 10 monclient(hunting): handle_auth_done global_id 24653 payload 274 2024-09-13T11:59:08.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f5671aab640 1 -- 172.21.15.100:0/2498122087 >> v2:172.21.15.122:3300/0 conn(0x7f566c09cb10 msgr2=0x7f566c09cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:08.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f5663fff640 1 -- 172.21.15.100:0/2498122087 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2835851898 0 0) 0x7f566c09e6e0 con 0x7f566c09f3a0 2024-09-13T11:59:08.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f5671aab640 1 --2- 172.21.15.100:0/2498122087 >> v2:172.21.15.122:3300/0 conn(0x7f566c09cb10 0x7f566c09cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f5671aab640 1 -- 172.21.15.100:0/2498122087 >> v1:172.21.15.100:6789/0 conn(0x7f566c09f3a0 legacy=0x7f566c09d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f5671aab640 10 monclient: _finish_hunting 0 2024-09-13T11:59:08.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f5671aab640 1 monclient: found mon.noname-b 2024-09-13T11:59:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f5671aab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f5671aab640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:59:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f5671aab640 1 -- 172.21.15.100:0/2498122087 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f566c0a1970 con 0x7f566c139530 2024-09-13T11:59:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.088+0000 7f5663fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:59:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.089+0000 7f5671aab640 1 --2- 172.21.15.100:0/2498122087 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f566c139530 0x7f566c13b920 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5664009d90 tx=0x7f566402a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bf22b016003876f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.089+0000 7f5663fff640 1 -- 172.21.15.100:0/2498122087 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5664031600 con 0x7f566c139530 2024-09-13T11:59:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.089+0000 7f5663fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.089+0000 7f5663fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:59:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.089+0000 7f5663fff640 10 monclient: dump: 2024-09-13T11:59:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.102 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.103 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:08.103 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.103 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:08.103 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:08.103 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:08.103 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.089+0000 7f5663fff640 10 monclient: _finish_auth 0 2024-09-13T11:59:08.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.089+0000 7f5663fff640 10 monclient: _check_auth_tickets 2024-09-13T11:59:08.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.089+0000 7f5663fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:08.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.089+0000 7f5663fff640 1 -- 172.21.15.100:0/2498122087 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56640317a0 con 0x7f566c139530 2024-09-13T11:59:08.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.089+0000 7f567350e640 5 monclient: authenticate success, global_id 24653 2024-09-13T11:59:08.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.089+0000 7f5663fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:08.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.089+0000 7f567350e640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:08.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.089+0000 7f567350e640 4 set_mon_vals no callback set 2024-09-13T11:59:08.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.089+0000 7f5663fff640 1 -- 172.21.15.100:0/2498122087 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5664031b70 con 0x7f566c139530 2024-09-13T11:59:08.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.089+0000 7f5663fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:08.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.089+0000 7f5663fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:59:08.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.089+0000 7f5663fff640 10 monclient: dump: 2024-09-13T11:59:08.104 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:08.104 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.104 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.104 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.104 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:08.104 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.104 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:08.104 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:08.104 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:08.104 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.090+0000 7f567350e640 10 monclient: shutdown 2024-09-13T11:59:08.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.090+0000 7f567350e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:08.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.090+0000 7f567350e640 1 -- 172.21.15.100:0/2498122087 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f566c139530 msgr2=0x7f566c13b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.090+0000 7f567350e640 1 --2- 172.21.15.100:0/2498122087 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f566c139530 0x7f566c13b920 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5664009d90 tx=0x7f566402a5d0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.090+0000 7f567350e640 1 -- 172.21.15.100:0/2498122087 shutdown_connections 2024-09-13T11:59:08.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.090+0000 7f567350e640 1 --2- 172.21.15.100:0/2498122087 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f566c139530 0x7f566c13b920 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.090+0000 7f567350e640 1 --2- 172.21.15.100:0/2498122087 >> v2:172.21.15.122:3300/0 conn(0x7f566c09cb10 0x7f566c09cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.090+0000 7f567350e640 1 -- 172.21.15.100:0/2498122087 >> 172.21.15.100:0/2498122087 conn(0x7f566c0a2620 msgr2=0x7f566c0a2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:08.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.090+0000 7f567350e640 1 -- 172.21.15.100:0/2498122087 shutdown_connections 2024-09-13T11:59:08.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.090+0000 7f567350e640 1 -- 172.21.15.100:0/2498122087 wait complete. 2024-09-13T11:59:08.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.090+0000 7f567350e640 10 monclient: build_initial_monmap 2024-09-13T11:59:08.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.090+0000 7f567350e640 10 monclient: monmap: 2024-09-13T11:59:08.105 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:08.106 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:08.106 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:08.106 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:08.106 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:08.106 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.106 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:08.106 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:08.106 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:08.106 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f567350e640 1 Processor -- start 2024-09-13T11:59:08.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f567350e640 1 -- start start 2024-09-13T11:59:08.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f567350e640 10 monclient: init 2024-09-13T11:59:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f567350e640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f567350e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:59:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f567350e640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f566c09cb10 0x7f566c135a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f567350e640 10 monclient(hunting): picked mon.noname-a con 0x7f566c09cb10 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f567350e640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f566c09f3a0 0x7f566c1c4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f567350e640 10 monclient(hunting): picked mon.noname-c con 0x7f566c09f3a0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f567350e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f566c139530 0x7f566c1c4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f567350e640 10 monclient(hunting): picked mon.noname-b con 0x7f566c139530 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f567350e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f567350e640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f566c0a1e10 con 0x7f566c09cb10 2024-09-13T11:59:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f567350e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f567350e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f566c0a1c90 con 0x7f566c139530 2024-09-13T11:59:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f567350e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:08.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f567350e640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f566c09ba00 con 0x7f566c09f3a0 2024-09-13T11:59:08.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f567350e640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:08.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f567350e640 10 monclient(hunting): authenticate will time out at 1128.685537s 2024-09-13T11:59:08.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f56712aa640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f566c09cb10 0x7f566c135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f56712aa640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f566c09cb10 0x7f566c135a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47244/0 (socket says 172.21.15.100:47244) 2024-09-13T11:59:08.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f5670aa9640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f566c09f3a0 0x7f566c1c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f56712aa640 1 -- 172.21.15.100:0/4174231858 learned_addr learned my addr 172.21.15.100:0/4174231858 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:08.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f56712aa640 10 monclient(hunting): get_auth_request con 0x7f566c09cb10 auth_method 0 2024-09-13T11:59:08.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f56712aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:08.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f56712aa640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f56712aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f5670aa9640 10 monclient(hunting): get_auth_request con 0x7f566c09f3a0 auth_method 0 2024-09-13T11:59:08.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f5670aa9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:08.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f5670aa9640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f5670aa9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f56712aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:08.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f56712aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:08.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f56712aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:08.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f5670aa9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:08.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f5670aa9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:08.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f5671aab640 1 --2- 172.21.15.100:0/4174231858 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f566c139530 0x7f566c1c4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f5670aa9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:08.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f5671aab640 10 monclient(hunting): get_auth_request con 0x7f566c139530 auth_method 0 2024-09-13T11:59:08.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f5671aab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:08.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f5671aab640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f5671aab640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f5671aab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:08.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f5671aab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:08.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f5671aab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:08.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f56712aa640 10 monclient(hunting): handle_auth_done global_id 15567 payload 824 2024-09-13T11:59:08.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f56712aa640 1 -- 172.21.15.100:0/4174231858 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f566c09f3a0 msgr2=0x7f566c1c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f56712aa640 1 --2- 172.21.15.100:0/4174231858 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f566c09f3a0 0x7f566c1c4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f56712aa640 1 -- 172.21.15.100:0/4174231858 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f566c139530 msgr2=0x7f566c1c4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f56712aa640 1 --2- 172.21.15.100:0/4174231858 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f566c139530 0x7f566c1c4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f56712aa640 10 monclient: _finish_hunting 0 2024-09-13T11:59:08.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f56712aa640 1 monclient: found mon.noname-a 2024-09-13T11:59:08.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f56712aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:08.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f56712aa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:08.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f56712aa640 1 -- 172.21.15.100:0/4174231858 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f566c120480 con 0x7f566c09cb10 2024-09-13T11:59:08.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.091+0000 7f5670aa9640 1 --2- 172.21.15.100:0/4174231858 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f566c09f3a0 0x7f566c1c4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:59:08.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.092+0000 7f56712aa640 1 --2- 172.21.15.100:0/4174231858 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f566c09cb10 0x7f566c135a50 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f566800e9b0 tx=0x7f566800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:08.111 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.092+0000 7f5661ffb640 1 -- 172.21.15.100:0/4174231858 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5668017070 con 0x7f566c09cb10 2024-09-13T11:59:08.111 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.092+0000 7f5661ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:08.111 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.092+0000 7f5661ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:08.111 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.092+0000 7f5661ffb640 10 monclient: dump: 2024-09-13T11:59:08.111 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:08.111 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.111 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.111 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.111 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:08.111 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.111 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:08.111 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:08.111 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:08.112 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.112 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.092+0000 7f5661ffb640 10 monclient: _finish_auth 0 2024-09-13T11:59:08.112 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.092+0000 7f5661ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:59:08.112 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.092+0000 7f5661ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:08.112 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.092+0000 7f5661ffb640 1 -- 172.21.15.100:0/4174231858 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f566800c9a0 con 0x7f566c09cb10 2024-09-13T11:59:08.112 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.092+0000 7f567350e640 5 monclient: authenticate success, global_id 15567 2024-09-13T11:59:08.112 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.092+0000 7f567350e640 10 monclient: _renew_subs 2024-09-13T11:59:08.112 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.092+0000 7f567350e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:08.112 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.092+0000 7f567350e640 1 -- 172.21.15.100:0/4174231858 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f566c120b50 con 0x7f566c09cb10 2024-09-13T11:59:08.112 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.092+0000 7f5661ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:08.112 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.092+0000 7f5661ffb640 1 -- 172.21.15.100:0/4174231858 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566800cd90 con 0x7f566c09cb10 2024-09-13T11:59:08.112 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.092+0000 7f5661ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:08.112 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.092+0000 7f5661ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.092+0000 7f5661ffb640 10 monclient: dump: 2024-09-13T11:59:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:08.113 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.092+0000 7f567350e640 10 monclient: _renew_subs 2024-09-13T11:59:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.092+0000 7f567350e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:08.113 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.092+0000 7f567350e640 1 -- 172.21.15.100:0/4174231858 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f566c1c51e0 con 0x7f566c09cb10 2024-09-13T11:59:08.114 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.092+0000 7f56637fe640 4 set_mon_vals no callback set 2024-09-13T11:59:08.114 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.093+0000 7f567350e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:08.114 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.093+0000 7f567350e640 1 -- 172.21.15.100:0/4174231858 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f566c09cef0 con 0x7f566c09cb10 2024-09-13T11:59:08.114 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.093+0000 7f5661ffb640 1 -- 172.21.15.100:0/4174231858 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f566801a070 con 0x7f566c09cb10 2024-09-13T11:59:08.114 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.093+0000 7f5661ffb640 1 --2- 172.21.15.100:0/4174231858 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f565003dfa0 0x7f5650040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.114 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.093+0000 7f5661ffb640 1 -- 172.21.15.100:0/4174231858 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7f56680525e0 con 0x7f566c09cb10 2024-09-13T11:59:08.114 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.093+0000 7f5661ffb640 1 --2- 172.21.15.100:0/4174231858 >> [v2:172.21.15.122:6808/3206511700,v1:172.21.15.122:6809/3206511700] conn(0x7f5650043e10 0x7f5650046230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.114 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.093+0000 7f5661ffb640 1 -- 172.21.15.100:0/4174231858 --> [v2:172.21.15.122:6808/3206511700,v1:172.21.15.122:6809/3206511700] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5668006870 con 0x7f5650043e10 2024-09-13T11:59:08.114 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.093+0000 7f5661ffb640 1 -- 172.21.15.100:0/4174231858 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_get_version_reply(handle=1 version=313) ==== 24+0+0 (secure 0 0 0) 0x7f5668059050 con 0x7f566c09cb10 2024-09-13T11:59:08.114 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.093+0000 7f5661ffb640 10 monclient: handle_get_version_reply finishing 1 version 313 2024-09-13T11:59:08.114 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.093+0000 7f5670aa9640 1 --2- 172.21.15.100:0/4174231858 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f565003dfa0 0x7f5650040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.114 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.093+0000 7f5670aa9640 10 monclient: get_auth_request con 0x7f565003dfa0 auth_method 0 2024-09-13T11:59:08.114 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.093+0000 7f5671aab640 1 --2- 172.21.15.100:0/4174231858 >> [v2:172.21.15.122:6808/3206511700,v1:172.21.15.122:6809/3206511700] conn(0x7f5650043e10 0x7f5650046230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.115 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.093+0000 7f5671aab640 10 monclient: get_auth_request con 0x7f5650043e10 auth_method 0 2024-09-13T11:59:08.115 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.094+0000 7f5670aa9640 1 --2- 172.21.15.100:0/4174231858 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f565003dfa0 0x7f5650040460 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f566c074080 tx=0x7f565c007cf0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:08.115 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.094+0000 7f5671aab640 1 --2- 172.21.15.100:0/4174231858 >> [v2:172.21.15.122:6808/3206511700,v1:172.21.15.122:6809/3206511700] conn(0x7f5650043e10 0x7f5650046230 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:08.115 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.095+0000 7f5661ffb640 1 -- 172.21.15.100:0/4174231858 <== osd.3 v2:172.21.15.122:6808/3206511700 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5668006870 con 0x7f5650043e10 2024-09-13T11:59:08.115 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.107+0000 7f567350e640 1 -- 172.21.15.100:0/4174231858 --> [v2:172.21.15.122:6808/3206511700,v1:172.21.15.122:6809/3206511700] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f566c121230 con 0x7f5650043e10 2024-09-13T11:59:08.115 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.107+0000 7f5661ffb640 1 -- 172.21.15.100:0/4174231858 <== osd.3 v2:172.21.15.122:6808/3206511700 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f5664003b60 con 0x7f5650043e10 2024-09-13T11:59:08.116 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:59:08.116 INFO:teuthology.orchestra.run.smithi100.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:59:08.116 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:59:08.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.107+0000 7f567350e640 1 -- 172.21.15.100:0/4174231858 >> [v2:172.21.15.122:6808/3206511700,v1:172.21.15.122:6809/3206511700] conn(0x7f5650043e10 msgr2=0x7f5650046230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.107+0000 7f567350e640 1 --2- 172.21.15.100:0/4174231858 >> [v2:172.21.15.122:6808/3206511700,v1:172.21.15.122:6809/3206511700] conn(0x7f5650043e10 0x7f5650046230 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.108+0000 7f567350e640 1 -- 172.21.15.100:0/4174231858 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f565003dfa0 msgr2=0x7f5650040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.108+0000 7f567350e640 1 --2- 172.21.15.100:0/4174231858 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f565003dfa0 0x7f5650040460 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f566c074080 tx=0x7f565c007cf0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.108+0000 7f567350e640 10 monclient: shutdown 2024-09-13T11:59:08.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.108+0000 7f567350e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:08.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.108+0000 7f567350e640 1 -- 172.21.15.100:0/4174231858 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f566c09cb10 msgr2=0x7f566c135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.117 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.108+0000 7f567350e640 1 --2- 172.21.15.100:0/4174231858 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f566c09cb10 0x7f566c135a50 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f566800e9b0 tx=0x7f566800ee70 comp rx=0 tx=0).stop 2024-09-13T11:59:08.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.108+0000 7f5671aab640 1 -- 172.21.15.100:0/4174231858 reap_dead start 2024-09-13T11:59:08.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.108+0000 7f567350e640 1 -- 172.21.15.100:0/4174231858 shutdown_connections 2024-09-13T11:59:08.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.108+0000 7f567350e640 1 -- 172.21.15.100:0/4174231858 >> 172.21.15.100:0/4174231858 conn(0x7f566c0a2620 msgr2=0x7f566c1384b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:08.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.108+0000 7f567350e640 1 -- 172.21.15.100:0/4174231858 shutdown_connections 2024-09-13T11:59:08.118 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.108+0000 7f567350e640 1 -- 172.21.15.100:0/4174231858 wait complete. 2024-09-13T11:59:08.120 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-13T11:59:08.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.215+0000 7fcc20aa2640 10 monclient: get_monmap_and_config 2024-09-13T11:59:08.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.215+0000 7fcc20aa2640 10 monclient: build_initial_monmap 2024-09-13T11:59:08.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.215+0000 7fcc20aa2640 10 monclient: monmap: 2024-09-13T11:59:08.217 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:08.218 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.218 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:08.217293+0000 2024-09-13T11:59:08.218 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:08.217293+0000 2024-09-13T11:59:08.218 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:08.218 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.218 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:08.218 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:08.218 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:08.218 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:08.218 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:08.218 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.215+0000 7fcc20aa2640 1 Processor -- start 2024-09-13T11:59:08.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc20aa2640 1 -- start start 2024-09-13T11:59:08.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc20aa2640 10 monclient: init 2024-09-13T11:59:08.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc20aa2640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:08.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc20aa2640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-13T11:59:08.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc20aa2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fcc1c0a15a0 0x7fcc1c0a1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc20aa2640 10 monclient(hunting): picked mon.noname-c con 0x7fcc1c0a15a0 addr v2:172.21.15.122:3300/0 2024-09-13T11:59:08.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc20aa2640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc1c09f6d0 0x7fcc1c0a1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc20aa2640 10 monclient(hunting): picked mon.noname-b con 0x7fcc1c09f6d0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:08.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc20aa2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcc1c1343d0 addr v1:172.21.15.100:6789/0 2024-09-13T11:59:08.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc20aa2640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc1c09ecf0 con 0x7fcc1c1343d0 2024-09-13T11:59:08.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc20aa2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:08.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc20aa2640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7fcc1c0a2400 con 0x7fcc1c0a15a0 2024-09-13T11:59:08.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc20aa2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:08.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc20aa2640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fcc1c09d3b0 con 0x7fcc1c09f6d0 2024-09-13T11:59:08.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc20aa2640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:08.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc20aa2640 10 monclient(hunting): authenticate will time out at 1128.810710s 2024-09-13T11:59:08.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc19d9b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc1c09f6d0 0x7fcc1c0a1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc1ad9d640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fcc1c1343d0 0x7fcc1c134860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:57140/0 (socket says 172.21.15.100:57140) 2024-09-13T11:59:08.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc1ad9d640 1 -- 172.21.15.100:0/1695186850 learned_addr learned my addr 172.21.15.100:0/1695186850 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:59:08.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc19d9b640 10 monclient(hunting): get_auth_request con 0x7fcc1c09f6d0 auth_method 0 2024-09-13T11:59:08.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc19d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:08.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc19d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc19d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc19d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:08.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc19d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:08.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.216+0000 7fcc19d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:08.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc1a59c640 1 --2- 172.21.15.100:0/1695186850 >> v2:172.21.15.122:3300/0 conn(0x7fcc1c0a15a0 0x7fcc1c0a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc1a59c640 10 monclient(hunting): get_auth_request con 0x7fcc1c0a15a0 auth_method 0 2024-09-13T11:59:08.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc1a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:08.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc1a59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc1a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc19d9b640 10 monclient(hunting): handle_auth_done global_id 24659 payload 274 2024-09-13T11:59:08.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc1959a640 1 -- 172.21.15.100:0/1695186850 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2376080380 0 0) 0x7fcc1c09ecf0 con 0x7fcc1c1343d0 2024-09-13T11:59:08.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc19d9b640 1 -- 172.21.15.100:0/1695186850 >> v2:172.21.15.122:3300/0 conn(0x7fcc1c0a15a0 msgr2=0x7fcc1c0a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:08.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc19d9b640 1 --2- 172.21.15.100:0/1695186850 >> v2:172.21.15.122:3300/0 conn(0x7fcc1c0a15a0 0x7fcc1c0a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc19d9b640 1 -- 172.21.15.100:0/1695186850 >> v1:172.21.15.100:6789/0 conn(0x7fcc1c1343d0 legacy=0x7fcc1c134860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc19d9b640 10 monclient: _finish_hunting 0 2024-09-13T11:59:08.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc19d9b640 1 monclient: found mon.noname-b 2024-09-13T11:59:08.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc19d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc19d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:59:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc19d9b640 1 -- 172.21.15.100:0/1695186850 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc1c09ea10 con 0x7fcc1c09f6d0 2024-09-13T11:59:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc1959a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:59:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc19d9b640 1 --2- 172.21.15.100:0/1695186850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc1c09f6d0 0x7fcc1c0a1ec0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009d90 tx=0x7fcc1002a530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=462c50b1f61bb808 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc1959a640 1 -- 172.21.15.100:0/1695186850 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10007740 con 0x7fcc1c09f6d0 2024-09-13T11:59:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc1959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc1959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:59:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc1959a640 10 monclient: dump: 2024-09-13T11:59:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.230 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.231 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:08.231 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.231 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:08.231 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:08.231 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:08.231 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc1959a640 10 monclient: _finish_auth 0 2024-09-13T11:59:08.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc1959a640 10 monclient: _check_auth_tickets 2024-09-13T11:59:08.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc1959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:08.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc20aa2640 5 monclient: authenticate success, global_id 24659 2024-09-13T11:59:08.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc1959a640 1 -- 172.21.15.100:0/1695186850 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc100078e0 con 0x7fcc1c09f6d0 2024-09-13T11:59:08.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc20aa2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:08.231 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc1959a640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:08.232 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc20aa2640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:08.232 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc20aa2640 4 set_mon_vals no callback set 2024-09-13T11:59:08.232 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc1959a640 1 -- 172.21.15.100:0/1695186850 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10007cb0 con 0x7fcc1c09f6d0 2024-09-13T11:59:08.232 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc1959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:08.232 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc1959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:59:08.232 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.217+0000 7fcc1959a640 10 monclient: dump: 2024-09-13T11:59:08.232 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:08.232 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.232 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.232 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.232 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:08.232 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.233 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:08.233 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:08.233 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:08.233 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.233 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.218+0000 7fcc20aa2640 10 monclient: shutdown 2024-09-13T11:59:08.233 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.218+0000 7fcc20aa2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:08.233 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.218+0000 7fcc20aa2640 1 -- 172.21.15.100:0/1695186850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc1c09f6d0 msgr2=0x7fcc1c0a1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.233 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.218+0000 7fcc20aa2640 1 --2- 172.21.15.100:0/1695186850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc1c09f6d0 0x7fcc1c0a1ec0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009d90 tx=0x7fcc1002a530 comp rx=0 tx=0).stop 2024-09-13T11:59:08.233 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.218+0000 7fcc20aa2640 1 -- 172.21.15.100:0/1695186850 shutdown_connections 2024-09-13T11:59:08.233 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.218+0000 7fcc20aa2640 1 --2- 172.21.15.100:0/1695186850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc1c09f6d0 0x7fcc1c0a1ec0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.233 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.218+0000 7fcc20aa2640 1 --2- 172.21.15.100:0/1695186850 >> v2:172.21.15.122:3300/0 conn(0x7fcc1c0a15a0 0x7fcc1c0a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.233 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.218+0000 7fcc20aa2640 1 -- 172.21.15.100:0/1695186850 >> 172.21.15.100:0/1695186850 conn(0x7fcc1c0a6470 msgr2=0x7fcc1c0a2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:08.233 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.218+0000 7fcc20aa2640 1 -- 172.21.15.100:0/1695186850 shutdown_connections 2024-09-13T11:59:08.234 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.218+0000 7fcc20aa2640 1 -- 172.21.15.100:0/1695186850 wait complete. 2024-09-13T11:59:08.234 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.219+0000 7fcc20aa2640 10 monclient: build_initial_monmap 2024-09-13T11:59:08.234 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.219+0000 7fcc20aa2640 10 monclient: monmap: 2024-09-13T11:59:08.234 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:08.234 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:08.234 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:08.234 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:08.234 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:08.234 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.234 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:08.234 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:08.235 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:08.235 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.219+0000 7fcc20aa2640 1 Processor -- start 2024-09-13T11:59:08.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.219+0000 7fcc20aa2640 1 -- start start 2024-09-13T11:59:08.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.219+0000 7fcc20aa2640 10 monclient: init 2024-09-13T11:59:08.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc20aa2640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:08.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc20aa2640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:59:08.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc20aa2640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc1c09f6d0 0x7fcc1c122ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc20aa2640 10 monclient(hunting): picked mon.noname-c con 0x7fcc1c09f6d0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:08.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc20aa2640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc1c0a15a0 0x7fcc1c123410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc20aa2640 10 monclient(hunting): picked mon.noname-a con 0x7fcc1c0a15a0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:08.235 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc20aa2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc1c1343d0 0x7fcc1c123950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc20aa2640 10 monclient(hunting): picked mon.noname-b con 0x7fcc1c1343d0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:08.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc20aa2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:08.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc20aa2640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1c09d120 con 0x7fcc1c0a15a0 2024-09-13T11:59:08.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc20aa2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:08.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc20aa2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1c09cfa0 con 0x7fcc1c1343d0 2024-09-13T11:59:08.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc20aa2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:08.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc20aa2640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fcc1c134f70 con 0x7fcc1c09f6d0 2024-09-13T11:59:08.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc20aa2640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:08.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc20aa2640 10 monclient(hunting): authenticate will time out at 1128.814089s 2024-09-13T11:59:08.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1a59c640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc1c09f6d0 0x7fcc1c122ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1a59c640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc1c09f6d0 0x7fcc1c122ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:52092/0 (socket says 172.21.15.100:52092) 2024-09-13T11:59:08.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1a59c640 1 -- 172.21.15.100:0/3523921096 learned_addr learned my addr 172.21.15.100:0/3523921096 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:08.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1a59c640 10 monclient(hunting): get_auth_request con 0x7fcc1c09f6d0 auth_method 0 2024-09-13T11:59:08.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:08.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1a59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc19d9b640 1 --2- 172.21.15.100:0/3523921096 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc1c0a15a0 0x7fcc1c123410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc19d9b640 10 monclient(hunting): get_auth_request con 0x7fcc1c0a15a0 auth_method 0 2024-09-13T11:59:08.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc19d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:08.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc19d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc19d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:08.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:08.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1ad9d640 1 --2- 172.21.15.100:0/3523921096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc1c1343d0 0x7fcc1c123950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:08.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1ad9d640 10 monclient(hunting): get_auth_request con 0x7fcc1c1343d0 auth_method 0 2024-09-13T11:59:08.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:08.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc19d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:08.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc19d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:08.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc19d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:08.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:08.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:08.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:08.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1a59c640 10 monclient(hunting): handle_auth_done global_id 24662 payload 824 2024-09-13T11:59:08.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1a59c640 1 -- 172.21.15.100:0/3523921096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc1c1343d0 msgr2=0x7fcc1c123950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1a59c640 1 --2- 172.21.15.100:0/3523921096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc1c1343d0 0x7fcc1c123950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1a59c640 1 -- 172.21.15.100:0/3523921096 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc1c0a15a0 msgr2=0x7fcc1c123410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1a59c640 1 --2- 172.21.15.100:0/3523921096 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc1c0a15a0 0x7fcc1c123410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1a59c640 10 monclient: _finish_hunting 0 2024-09-13T11:59:08.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1a59c640 1 monclient: found mon.noname-c 2024-09-13T11:59:08.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:08.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:59:08.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1a59c640 1 -- 172.21.15.100:0/3523921096 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc1c1259a0 con 0x7fcc1c09f6d0 2024-09-13T11:59:08.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc19d9b640 1 --2- 172.21.15.100:0/3523921096 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc1c0a15a0 0x7fcc1c123410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:59:08.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc1a59c640 1 --2- 172.21.15.100:0/3523921096 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc1c09f6d0 0x7fcc1c122ed0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcc1400d800 tx=0x7fcc1400dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:08.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc0affd640 1 -- 172.21.15.100:0/3523921096 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1400b6c0 con 0x7fcc1c09f6d0 2024-09-13T11:59:08.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:08.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc0affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:59:08.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc0affd640 10 monclient: dump: 2024-09-13T11:59:08.240 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:08.240 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.240 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.240 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.240 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:08.240 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.240 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:08.240 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:08.240 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:08.240 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc0affd640 10 monclient: _finish_auth 0 2024-09-13T11:59:08.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc0affd640 10 monclient: _check_auth_tickets 2024-09-13T11:59:08.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:08.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc0affd640 1 -- 172.21.15.100:0/3523921096 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc14019070 con 0x7fcc1c09f6d0 2024-09-13T11:59:08.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc20aa2640 5 monclient: authenticate success, global_id 24662 2024-09-13T11:59:08.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc20aa2640 10 monclient: _renew_subs 2024-09-13T11:59:08.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc20aa2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:59:08.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc20aa2640 1 -- 172.21.15.100:0/3523921096 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc1c125c80 con 0x7fcc1c09f6d0 2024-09-13T11:59:08.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc0affd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:08.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc0affd640 1 -- 172.21.15.100:0/3523921096 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1400ba80 con 0x7fcc1c09f6d0 2024-09-13T11:59:08.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:08.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc0affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:59:08.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc0affd640 10 monclient: dump: 2024-09-13T11:59:08.241 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:08.242 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.242 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.242 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.242 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:08.242 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.242 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:08.242 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:08.242 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:08.242 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc18d99640 4 set_mon_vals no callback set 2024-09-13T11:59:08.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc20aa2640 10 monclient: _renew_subs 2024-09-13T11:59:08.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc20aa2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:59:08.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.220+0000 7fcc20aa2640 1 -- 172.21.15.100:0/3523921096 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc1c09c4f0 con 0x7fcc1c09f6d0 2024-09-13T11:59:08.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.221+0000 7fcc0affd640 1 -- 172.21.15.100:0/3523921096 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fcc1400bc20 con 0x7fcc1c09f6d0 2024-09-13T11:59:08.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.221+0000 7fcc20aa2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:08.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.221+0000 7fcc20aa2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:08.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.221+0000 7fcc20aa2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:59:08.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.221+0000 7fcc20aa2640 1 -- 172.21.15.100:0/3523921096 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc1c064510 con 0x7fcc1c09f6d0 2024-09-13T11:59:08.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.221+0000 7fcc0affd640 1 --2- 172.21.15.100:0/3523921096 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fcbfc03dfa0 0x7fcbfc040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.221+0000 7fcc0affd640 1 -- 172.21.15.100:0/3523921096 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7fcc140529c0 con 0x7fcc1c09f6d0 2024-09-13T11:59:08.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.222+0000 7fcc19d9b640 1 --2- 172.21.15.100:0/3523921096 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fcbfc03dfa0 0x7fcbfc040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.222+0000 7fcc19d9b640 10 monclient: get_auth_request con 0x7fcbfc03dfa0 auth_method 0 2024-09-13T11:59:08.244 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.222+0000 7fcc19d9b640 1 --2- 172.21.15.100:0/3523921096 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fcbfc03dfa0 0x7fcbfc040460 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fcc10034000 tx=0x7fcc1002f000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:08.244 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.226+0000 7fcc0affd640 1 -- 172.21.15.100:0/3523921096 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fcc1c064510 con 0x7fcc1c09f6d0 2024-09-13T11:59:08.244 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.226+0000 7fcc0affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:08.244 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.226+0000 7fcc0affd640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:59:08.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.400+0000 7fcc20aa2640 1 -- 172.21.15.100:0/3523921096 --> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fcc1c134860 con 0x7fcbfc03dfa0 2024-09-13T11:59:08.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.401+0000 7fcc0affd640 1 -- 172.21.15.100:0/3523921096 <== mgr.4114 v2:172.21.15.122:6824/441789830 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fcc1c134860 con 0x7fcbfc03dfa0 2024-09-13T11:59:08.403 INFO:teuthology.orchestra.run.smithi100.stderr:instructed osd(s) 3 to deep-scrub 2024-09-13T11:59:08.405 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.403+0000 7fcc20aa2640 1 -- 172.21.15.100:0/3523921096 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fcbfc03dfa0 msgr2=0x7fcbfc040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.405 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.403+0000 7fcc20aa2640 1 --2- 172.21.15.100:0/3523921096 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fcbfc03dfa0 0x7fcbfc040460 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fcc10034000 tx=0x7fcc1002f000 comp rx=0 tx=0).stop 2024-09-13T11:59:08.405 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.403+0000 7fcc20aa2640 10 monclient: shutdown 2024-09-13T11:59:08.405 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.403+0000 7fcc20aa2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:08.405 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.403+0000 7fcc20aa2640 1 -- 172.21.15.100:0/3523921096 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc1c09f6d0 msgr2=0x7fcc1c122ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.403+0000 7fcc20aa2640 1 --2- 172.21.15.100:0/3523921096 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc1c09f6d0 0x7fcc1c122ed0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcc1400d800 tx=0x7fcc1400dcc0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.403+0000 7fcc20aa2640 1 -- 172.21.15.100:0/3523921096 shutdown_connections 2024-09-13T11:59:08.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.403+0000 7fcc20aa2640 1 --2- 172.21.15.100:0/3523921096 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fcbfc03dfa0 0x7fcbfc040460 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.403+0000 7fcc20aa2640 1 --2- 172.21.15.100:0/3523921096 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc1c0a15a0 0x7fcc1c123410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.403+0000 7fcc20aa2640 1 --2- 172.21.15.100:0/3523921096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc1c1343d0 0x7fcc1c123950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.403+0000 7fcc20aa2640 1 --2- 172.21.15.100:0/3523921096 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcc1c09f6d0 0x7fcc1c122ed0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.403+0000 7fcc20aa2640 1 -- 172.21.15.100:0/3523921096 >> 172.21.15.100:0/3523921096 conn(0x7fcc1c0a6470 msgr2=0x7fcc1c071940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:08.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.403+0000 7fcc20aa2640 1 -- 172.21.15.100:0/3523921096 shutdown_connections 2024-09-13T11:59:08.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.403+0000 7fcc20aa2640 1 -- 172.21.15.100:0/3523921096 wait complete. 2024-09-13T11:59:08.416 INFO:tasks.ceph:Scrubbing osd.4 2024-09-13T11:59:08.416 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:59:08.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.515+0000 7fc3e1910640 10 monclient: get_monmap_and_config 2024-09-13T11:59:08.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.515+0000 7fc3e1910640 10 monclient: build_initial_monmap 2024-09-13T11:59:08.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.515+0000 7fc3e1910640 10 monclient: monmap: 2024-09-13T11:59:08.518 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:08.518 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.518 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:08.517703+0000 2024-09-13T11:59:08.518 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:08.517703+0000 2024-09-13T11:59:08.518 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:08.518 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.519 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:08.519 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:08.519 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:08.519 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:08.519 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:08.519 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.528 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.516+0000 7fc3e1910640 1 Processor -- start 2024-09-13T11:59:08.528 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.516+0000 7fc3e1910640 1 -- start start 2024-09-13T11:59:08.528 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.516+0000 7fc3e1910640 10 monclient: init 2024-09-13T11:59:08.528 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3e1910640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:08.528 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3e1910640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-13T11:59:08.528 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3e1910640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc3dc0a48e0 addr v1:172.21.15.122:6789/0 2024-09-13T11:59:08.528 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3e1910640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3dc0a53d0 0x7fc3dc0a28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.528 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3e1910640 10 monclient(hunting): picked mon.noname-b con 0x7fc3dc0a53d0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:08.528 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3e1910640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc3dc13caf0 0x7fc3dc13eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.529 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3e1910640 10 monclient(hunting): picked mon.noname-a con 0x7fc3dc13caf0 addr v2:172.21.15.100:3300/0 2024-09-13T11:59:08.529 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3e1910640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3dc070090 con 0x7fc3dc0a48e0 2024-09-13T11:59:08.529 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3e1910640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:08.529 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3e1910640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7fc3dc0a5a80 con 0x7fc3dc13caf0 2024-09-13T11:59:08.529 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3e1910640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:08.529 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3e1910640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fc3dc0a1bb0 con 0x7fc3dc0a53d0 2024-09-13T11:59:08.532 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3e1910640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:08.532 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3e1910640 10 monclient(hunting): authenticate will time out at 1129.111098s 2024-09-13T11:59:08.532 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3db7fe640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc3dc13caf0 0x7fc3dc13eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.533 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3da7fc640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3dc0a53d0 0x7fc3dc0a28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.533 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3db7fe640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc3dc13caf0 0x7fc3dc13eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47264/0 (socket says 172.21.15.100:47264) 2024-09-13T11:59:08.533 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3db7fe640 1 -- 172.21.15.100:0/3512605044 learned_addr learned my addr 172.21.15.100:0/3512605044 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:08.533 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3db7fe640 10 monclient(hunting): get_auth_request con 0x7fc3dc13caf0 auth_method 0 2024-09-13T11:59:08.533 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:08.533 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.533 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.533 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3da7fc640 10 monclient(hunting): get_auth_request con 0x7fc3dc0a53d0 auth_method 0 2024-09-13T11:59:08.533 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3da7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:08.533 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.533 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.533 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:08.533 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:08.534 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:08.534 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3da7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:08.534 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3da7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:08.534 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3da7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:08.534 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3db7fe640 10 monclient(hunting): handle_auth_done global_id 15576 payload 274 2024-09-13T11:59:08.534 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3db7fe640 1 -- 172.21.15.100:0/3512605044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3dc0a53d0 msgr2=0x7fc3dc0a28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:08.534 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3db7fe640 1 --2- 172.21.15.100:0/3512605044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3dc0a53d0 0x7fc3dc0a28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.534 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3db7fe640 1 -- 172.21.15.100:0/3512605044 >> v1:172.21.15.122:6789/0 conn(0x7fc3dc0a48e0 legacy=0x7fc3dc0a4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.534 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3db7fe640 10 monclient: _finish_hunting 0 2024-09-13T11:59:08.534 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3db7fe640 1 monclient: found mon.noname-a 2024-09-13T11:59:08.534 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:08.534 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3db7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:08.535 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3db7fe640 1 -- 172.21.15.100:0/3512605044 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3dc0a15f0 con 0x7fc3dc13caf0 2024-09-13T11:59:08.535 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.517+0000 7fc3da7fc640 1 --2- 172.21.15.100:0/3512605044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3dc0a53d0 0x7fc3dc0a28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:59:08.535 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3db7fe640 1 --2- 172.21.15.100:0/3512605044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc3dc13caf0 0x7fc3dc13eee0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4009e50 tx=0x7fc3d402a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adbfa08e488fd275 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:08.535 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3d9ffb640 1 -- 172.21.15.100:0/3512605044 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d4036070 con 0x7fc3dc13caf0 2024-09-13T11:59:08.535 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:08.535 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3d9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:08.535 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3d9ffb640 10 monclient: dump: 2024-09-13T11:59:08.535 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:08.535 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.535 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.535 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.535 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:08.535 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.535 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:08.536 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:08.536 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:08.536 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.536 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3d9ffb640 10 monclient: _finish_auth 0 2024-09-13T11:59:08.536 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3d9ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:59:08.536 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:08.536 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3d9ffb640 1 -- 172.21.15.100:0/3512605044 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3d4031600 con 0x7fc3dc13caf0 2024-09-13T11:59:08.536 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3e1910640 5 monclient: authenticate success, global_id 15576 2024-09-13T11:59:08.536 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3e1910640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:08.536 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3d9ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:08.536 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3e1910640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:08.536 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3d9ffb640 1 -- 172.21.15.100:0/3512605044 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d40319d0 con 0x7fc3dc13caf0 2024-09-13T11:59:08.537 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3e1910640 4 set_mon_vals no callback set 2024-09-13T11:59:08.537 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:08.537 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3d9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:08.537 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3d9ffb640 10 monclient: dump: 2024-09-13T11:59:08.537 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:08.537 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.537 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.537 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.537 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:08.537 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.537 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:08.538 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:08.538 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:08.538 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.538 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3e1910640 10 monclient: shutdown 2024-09-13T11:59:08.538 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3e1910640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:08.538 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3e1910640 1 -- 172.21.15.100:0/3512605044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc3dc13caf0 msgr2=0x7fc3dc13eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.538 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3e1910640 1 --2- 172.21.15.100:0/3512605044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc3dc13caf0 0x7fc3dc13eee0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4009e50 tx=0x7fc3d402a5f0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.538 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3e1910640 1 -- 172.21.15.100:0/3512605044 shutdown_connections 2024-09-13T11:59:08.538 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3e1910640 1 --2- 172.21.15.100:0/3512605044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc3dc13caf0 0x7fc3dc13eee0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.538 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3e1910640 1 --2- 172.21.15.100:0/3512605044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3dc0a53d0 0x7fc3dc0a28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.538 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3e1910640 1 -- 172.21.15.100:0/3512605044 >> 172.21.15.100:0/3512605044 conn(0x7fc3dc0a69e0 msgr2=0x7fc3dc0a6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:08.538 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3e1910640 1 -- 172.21.15.100:0/3512605044 shutdown_connections 2024-09-13T11:59:08.538 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.518+0000 7fc3e1910640 1 -- 172.21.15.100:0/3512605044 wait complete. 2024-09-13T11:59:08.539 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.519+0000 7fc3e1910640 10 monclient: build_initial_monmap 2024-09-13T11:59:08.539 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.519+0000 7fc3e1910640 10 monclient: monmap: 2024-09-13T11:59:08.539 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:08.539 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:08.539 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:08.539 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:08.539 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:08.539 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.539 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:08.539 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:08.539 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:08.539 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.519+0000 7fc3e1910640 1 Processor -- start 2024-09-13T11:59:08.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.519+0000 7fc3e1910640 1 -- start start 2024-09-13T11:59:08.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.519+0000 7fc3e1910640 10 monclient: init 2024-09-13T11:59:08.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3e1910640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:08.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3e1910640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:59:08.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3e1910640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3dc0a48e0 0x7fc3dc137780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3e1910640 10 monclient(hunting): picked mon.noname-c con 0x7fc3dc0a48e0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:08.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3e1910640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3dc0a53d0 0x7fc3dc09ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3e1910640 10 monclient(hunting): picked mon.noname-b con 0x7fc3dc0a53d0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:08.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3e1910640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc3dc13caf0 0x7fc3dc09f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3e1910640 10 monclient(hunting): picked mon.noname-a con 0x7fc3dc13caf0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:08.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3e1910640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:08.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3e1910640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fc3dc06fcd0 con 0x7fc3dc13caf0 2024-09-13T11:59:08.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3e1910640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:08.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3e1910640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc3dc0a0f10 con 0x7fc3dc0a53d0 2024-09-13T11:59:08.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3e1910640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:08.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3e1910640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fc3dc06fe50 con 0x7fc3dc0a48e0 2024-09-13T11:59:08.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3e1910640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:08.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3e1910640 10 monclient(hunting): authenticate will time out at 1129.114398s 2024-09-13T11:59:08.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3daffd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3dc0a48e0 0x7fc3dc137780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3db7fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc3dc13caf0 0x7fc3dc09f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3daffd640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3dc0a48e0 0x7fc3dc137780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:52106/0 (socket says 172.21.15.100:52106) 2024-09-13T11:59:08.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3daffd640 1 -- 172.21.15.100:0/1511528540 learned_addr learned my addr 172.21.15.100:0/1511528540 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:08.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3db7fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc3dc13caf0 0x7fc3dc09f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47276/0 (socket says 172.21.15.100:47276) 2024-09-13T11:59:08.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3daffd640 10 monclient(hunting): get_auth_request con 0x7fc3dc0a48e0 auth_method 0 2024-09-13T11:59:08.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:08.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3daffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3db7fe640 10 monclient(hunting): get_auth_request con 0x7fc3dc13caf0 auth_method 0 2024-09-13T11:59:08.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:08.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3da7fc640 1 --2- 172.21.15.100:0/1511528540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3dc0a53d0 0x7fc3dc09ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3da7fc640 10 monclient(hunting): get_auth_request con 0x7fc3dc0a53d0 auth_method 0 2024-09-13T11:59:08.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3da7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:08.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:08.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:08.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:08.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:08.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:08.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:08.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3da7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:08.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3da7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:08.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3da7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:08.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3daffd640 10 monclient(hunting): handle_auth_done global_id 24668 payload 824 2024-09-13T11:59:08.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3daffd640 1 -- 172.21.15.100:0/1511528540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3dc0a53d0 msgr2=0x7fc3dc09ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3daffd640 1 --2- 172.21.15.100:0/1511528540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3dc0a53d0 0x7fc3dc09ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3daffd640 1 -- 172.21.15.100:0/1511528540 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc3dc13caf0 msgr2=0x7fc3dc09f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3daffd640 1 --2- 172.21.15.100:0/1511528540 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc3dc13caf0 0x7fc3dc09f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3daffd640 10 monclient: _finish_hunting 0 2024-09-13T11:59:08.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3daffd640 1 monclient: found mon.noname-c 2024-09-13T11:59:08.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:08.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3daffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:59:08.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3daffd640 1 -- 172.21.15.100:0/1511528540 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3dc0a10d0 con 0x7fc3dc0a48e0 2024-09-13T11:59:08.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3db7fe640 1 --2- 172.21.15.100:0/1511528540 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc3dc13caf0 0x7fc3dc09f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:59:08.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.520+0000 7fc3daffd640 1 --2- 172.21.15.100:0/1511528540 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3dc0a48e0 0x7fc3dc137780 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc00c450 tx=0x7fc3cc00c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:08.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.521+0000 7fc3d3fff640 1 -- 172.21.15.100:0/1511528540 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc0052d0 con 0x7fc3dc0a48e0 2024-09-13T11:59:08.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.521+0000 7fc3d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:08.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.521+0000 7fc3d3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:59:08.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.521+0000 7fc3d3fff640 10 monclient: dump: 2024-09-13T11:59:08.545 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:08.545 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.545 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.545 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.545 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:08.545 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.545 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:08.545 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:08.546 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:08.546 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.546 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.521+0000 7fc3d3fff640 10 monclient: _finish_auth 0 2024-09-13T11:59:08.546 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.521+0000 7fc3d3fff640 10 monclient: _check_auth_tickets 2024-09-13T11:59:08.546 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.521+0000 7fc3d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:08.546 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.521+0000 7fc3d3fff640 1 -- 172.21.15.100:0/1511528540 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3cc005470 con 0x7fc3dc0a48e0 2024-09-13T11:59:08.546 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.521+0000 7fc3e1910640 5 monclient: authenticate success, global_id 24668 2024-09-13T11:59:08.546 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.521+0000 7fc3e1910640 10 monclient: _renew_subs 2024-09-13T11:59:08.546 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.521+0000 7fc3e1910640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:59:08.547 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.521+0000 7fc3e1910640 1 -- 172.21.15.100:0/1511528540 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3dc0a0500 con 0x7fc3dc0a48e0 2024-09-13T11:59:08.547 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.521+0000 7fc3d3fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:08.547 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.521+0000 7fc3d3fff640 1 -- 172.21.15.100:0/1511528540 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc012870 con 0x7fc3dc0a48e0 2024-09-13T11:59:08.547 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.521+0000 7fc3d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:08.547 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.521+0000 7fc3d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:59:08.547 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.521+0000 7fc3d3fff640 10 monclient: dump: 2024-09-13T11:59:08.547 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:08.547 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.547 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.548 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.548 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:08.548 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.548 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:08.548 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:08.548 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:08.548 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.548 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.521+0000 7fc3d97fa640 4 set_mon_vals no callback set 2024-09-13T11:59:08.548 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.521+0000 7fc3e1910640 10 monclient: _renew_subs 2024-09-13T11:59:08.548 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.521+0000 7fc3e1910640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:59:08.548 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.521+0000 7fc3e1910640 1 -- 172.21.15.100:0/1511528540 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3dc120530 con 0x7fc3dc0a48e0 2024-09-13T11:59:08.548 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.521+0000 7fc3d3fff640 1 -- 172.21.15.100:0/1511528540 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fc3cc012a10 con 0x7fc3dc0a48e0 2024-09-13T11:59:08.549 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.522+0000 7fc3e1910640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:59:08.549 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.522+0000 7fc3e1910640 1 -- 172.21.15.100:0/1511528540 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc3dc0a28c0 con 0x7fc3dc0a48e0 2024-09-13T11:59:08.549 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.522+0000 7fc3d3fff640 1 --2- 172.21.15.100:0/1511528540 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc3bc03dfa0 0x7fc3bc040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.549 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.522+0000 7fc3d3fff640 1 -- 172.21.15.100:0/1511528540 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7fc3cc0520b0 con 0x7fc3dc0a48e0 2024-09-13T11:59:08.549 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.522+0000 7fc3d3fff640 1 --2- 172.21.15.100:0/1511528540 >> [v2:172.21.15.122:6816/777240702,v1:172.21.15.122:6817/777240702] conn(0x7fc3bc043e10 0x7fc3bc046230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.549 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.522+0000 7fc3d3fff640 1 -- 172.21.15.100:0/1511528540 --> [v2:172.21.15.122:6816/777240702,v1:172.21.15.122:6817/777240702] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc3bc0468e0 con 0x7fc3bc043e10 2024-09-13T11:59:08.549 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.522+0000 7fc3d3fff640 1 -- 172.21.15.100:0/1511528540 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_get_version_reply(handle=1 version=313) ==== 24+0+0 (secure 0 0 0) 0x7fc3cc059050 con 0x7fc3dc0a48e0 2024-09-13T11:59:08.549 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.522+0000 7fc3d3fff640 10 monclient: handle_get_version_reply finishing 1 version 313 2024-09-13T11:59:08.549 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.522+0000 7fc3da7fc640 1 --2- 172.21.15.100:0/1511528540 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc3bc03dfa0 0x7fc3bc040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.549 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.522+0000 7fc3da7fc640 10 monclient: get_auth_request con 0x7fc3bc03dfa0 auth_method 0 2024-09-13T11:59:08.549 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.522+0000 7fc3db7fe640 1 --2- 172.21.15.100:0/1511528540 >> [v2:172.21.15.122:6816/777240702,v1:172.21.15.122:6817/777240702] conn(0x7fc3bc043e10 0x7fc3bc046230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.549 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.522+0000 7fc3db7fe640 10 monclient: get_auth_request con 0x7fc3bc043e10 auth_method 0 2024-09-13T11:59:08.549 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.522+0000 7fc3da7fc640 1 --2- 172.21.15.100:0/1511528540 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc3bc03dfa0 0x7fc3bc040460 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4009560 tx=0x7fc3c4009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:08.549 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.523+0000 7fc3db7fe640 1 --2- 172.21.15.100:0/1511528540 >> [v2:172.21.15.122:6816/777240702,v1:172.21.15.122:6817/777240702] conn(0x7fc3bc043e10 0x7fc3bc046230 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:08.550 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.523+0000 7fc3d3fff640 1 -- 172.21.15.100:0/1511528540 <== osd.4 v2:172.21.15.122:6816/777240702 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc3bc0468e0 con 0x7fc3bc043e10 2024-09-13T11:59:08.550 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.536+0000 7fc3e1910640 1 -- 172.21.15.100:0/1511528540 --> [v2:172.21.15.122:6816/777240702,v1:172.21.15.122:6817/777240702] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc3dc121940 con 0x7fc3bc043e10 2024-09-13T11:59:08.550 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.536+0000 7fc3d3fff640 1 -- 172.21.15.100:0/1511528540 <== osd.4 v2:172.21.15.122:6816/777240702 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc3dc121940 con 0x7fc3bc043e10 2024-09-13T11:59:08.551 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:59:08.551 INFO:teuthology.orchestra.run.smithi100.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:59:08.551 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:59:08.552 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.537+0000 7fc3e1910640 1 -- 172.21.15.100:0/1511528540 >> [v2:172.21.15.122:6816/777240702,v1:172.21.15.122:6817/777240702] conn(0x7fc3bc043e10 msgr2=0x7fc3bc046230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.537+0000 7fc3e1910640 1 --2- 172.21.15.100:0/1511528540 >> [v2:172.21.15.122:6816/777240702,v1:172.21.15.122:6817/777240702] conn(0x7fc3bc043e10 0x7fc3bc046230 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.537+0000 7fc3e1910640 1 -- 172.21.15.100:0/1511528540 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc3bc03dfa0 msgr2=0x7fc3bc040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.537+0000 7fc3e1910640 1 --2- 172.21.15.100:0/1511528540 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fc3bc03dfa0 0x7fc3bc040460 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4009560 tx=0x7fc3c4009190 comp rx=0 tx=0).stop 2024-09-13T11:59:08.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.537+0000 7fc3e1910640 10 monclient: shutdown 2024-09-13T11:59:08.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.537+0000 7fc3e1910640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:08.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.537+0000 7fc3e1910640 1 -- 172.21.15.100:0/1511528540 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3dc0a48e0 msgr2=0x7fc3dc137780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.537+0000 7fc3e1910640 1 --2- 172.21.15.100:0/1511528540 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc3dc0a48e0 0x7fc3dc137780 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc00c450 tx=0x7fc3cc00c910 comp rx=0 tx=0).stop 2024-09-13T11:59:08.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.537+0000 7fc3db7fe640 1 -- 172.21.15.100:0/1511528540 reap_dead start 2024-09-13T11:59:08.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.537+0000 7fc3e1910640 1 -- 172.21.15.100:0/1511528540 shutdown_connections 2024-09-13T11:59:08.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.537+0000 7fc3e1910640 1 -- 172.21.15.100:0/1511528540 >> 172.21.15.100:0/1511528540 conn(0x7fc3dc0a69e0 msgr2=0x7fc3dc0a9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:08.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.537+0000 7fc3e1910640 1 -- 172.21.15.100:0/1511528540 shutdown_connections 2024-09-13T11:59:08.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.537+0000 7fc3e1910640 1 -- 172.21.15.100:0/1511528540 wait complete. 2024-09-13T11:59:08.554 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-13T11:59:08.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.649+0000 7f71046a2640 10 monclient: get_monmap_and_config 2024-09-13T11:59:08.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.649+0000 7f71046a2640 10 monclient: build_initial_monmap 2024-09-13T11:59:08.652 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.649+0000 7f71046a2640 10 monclient: monmap: 2024-09-13T11:59:08.652 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:08.652 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.652 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:08.651092+0000 2024-09-13T11:59:08.652 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:08.651092+0000 2024-09-13T11:59:08.652 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:08.652 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.652 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:08.652 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:08.652 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:08.652 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:08.653 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:08.653 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.653 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.649+0000 7f71046a2640 1 Processor -- start 2024-09-13T11:59:08.653 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.649+0000 7f71046a2640 1 -- start start 2024-09-13T11:59:08.653 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.649+0000 7f71046a2640 10 monclient: init 2024-09-13T11:59:08.662 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f71046a2640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:08.662 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f71046a2640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-13T11:59:08.662 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f71046a2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f70fc0a15a0 0x7f70fc0a1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.662 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f71046a2640 10 monclient(hunting): picked mon.noname-c con 0x7f70fc0a15a0 addr v2:172.21.15.122:3300/0 2024-09-13T11:59:08.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f71046a2640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70fc09f6d0 0x7f70fc0a1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f71046a2640 10 monclient(hunting): picked mon.noname-b con 0x7f70fc09f6d0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:08.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f71046a2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f70fc1343d0 addr v1:172.21.15.100:6789/0 2024-09-13T11:59:08.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f71046a2640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70fc09ecf0 con 0x7f70fc1343d0 2024-09-13T11:59:08.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f71046a2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:08.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f71046a2640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f70fc0a2400 con 0x7f70fc0a15a0 2024-09-13T11:59:08.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f71046a2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:08.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f71046a2640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f70fc09d3b0 con 0x7f70fc09f6d0 2024-09-13T11:59:08.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f71046a2640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:08.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f71046a2640 10 monclient(hunting): authenticate will time out at 1129.244483s 2024-09-13T11:59:08.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f7101c3d640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70fc09f6d0 0x7f70fc0a1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f7102c3f640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f70fc1343d0 0x7f70fc134860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:57144/0 (socket says 172.21.15.100:57144) 2024-09-13T11:59:08.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f7102c3f640 1 -- 172.21.15.100:0/3977390282 learned_addr learned my addr 172.21.15.100:0/3977390282 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:59:08.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f7101c3d640 10 monclient(hunting): get_auth_request con 0x7f70fc09f6d0 auth_method 0 2024-09-13T11:59:08.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f7101c3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:08.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f7101c3d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f7101c3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f7101c3d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:08.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f7101c3d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:08.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f7101c3d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:08.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f710243e640 1 --2- 172.21.15.100:0/3977390282 >> v2:172.21.15.122:3300/0 conn(0x7f70fc0a15a0 0x7f70fc0a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f710243e640 10 monclient(hunting): get_auth_request con 0x7f70fc0a15a0 auth_method 0 2024-09-13T11:59:08.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f710243e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:08.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f710243e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f710243e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f710243e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:08.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f710243e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:08.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f710143c640 1 -- 172.21.15.100:0/3977390282 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3352239785 0 0) 0x7f70fc09ecf0 con 0x7f70fc1343d0 2024-09-13T11:59:08.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f710243e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:08.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f710143c640 10 monclient(hunting): _negotiate 2024-09-13T11:59:08.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f710143c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f710143c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f710143c640 10 monclient(hunting): my global_id is 15582 2024-09-13T11:59:08.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f710143c640 1 -- 172.21.15.100:0/3977390282 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70e8003520 con 0x7f70fc1343d0 2024-09-13T11:59:08.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f7101c3d640 10 monclient(hunting): handle_auth_done global_id 24671 payload 274 2024-09-13T11:59:08.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f7101c3d640 1 -- 172.21.15.100:0/3977390282 >> v2:172.21.15.122:3300/0 conn(0x7f70fc0a15a0 msgr2=0x7f70fc0a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:08.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f7101c3d640 1 --2- 172.21.15.100:0/3977390282 >> v2:172.21.15.122:3300/0 conn(0x7f70fc0a15a0 0x7f70fc0a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f7101c3d640 1 -- 172.21.15.100:0/3977390282 >> v1:172.21.15.100:6789/0 conn(0x7f70fc1343d0 legacy=0x7f70fc134860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f7101c3d640 10 monclient: _finish_hunting 0 2024-09-13T11:59:08.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f7101c3d640 1 monclient: found mon.noname-b 2024-09-13T11:59:08.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f7101c3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:08.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f7101c3d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:59:08.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.650+0000 7f7101c3d640 1 -- 172.21.15.100:0/3977390282 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70fc09ea10 con 0x7f70fc09f6d0 2024-09-13T11:59:08.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.651+0000 7f7101c3d640 1 --2- 172.21.15.100:0/3977390282 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70fc09f6d0 0x7f70fc0a1ec0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f70ec0097a0 tx=0x7f70ec02a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2145cad654f8c949 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:08.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.651+0000 7f710143c640 1 -- 172.21.15.100:0/3977390282 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ec037070 con 0x7f70fc09f6d0 2024-09-13T11:59:08.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.651+0000 7f710143c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:08.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.651+0000 7f710143c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:59:08.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.651+0000 7f710143c640 10 monclient: dump: 2024-09-13T11:59:08.667 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:08.667 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.667 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.667 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.667 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:08.667 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.667 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:08.667 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:08.667 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:08.667 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.651+0000 7f710143c640 10 monclient: _finish_auth 0 2024-09-13T11:59:08.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.651+0000 7f710143c640 10 monclient: _check_auth_tickets 2024-09-13T11:59:08.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.651+0000 7f710143c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:08.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.651+0000 7f710143c640 1 -- 172.21.15.100:0/3977390282 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70ec02adb0 con 0x7f70fc09f6d0 2024-09-13T11:59:08.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.651+0000 7f71046a2640 5 monclient: authenticate success, global_id 24671 2024-09-13T11:59:08.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.651+0000 7f71046a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:08.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.651+0000 7f710143c640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:08.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.651+0000 7f71046a2640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:08.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.651+0000 7f710143c640 1 -- 172.21.15.100:0/3977390282 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ec032660 con 0x7f70fc09f6d0 2024-09-13T11:59:08.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.651+0000 7f710143c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:08.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.651+0000 7f71046a2640 4 set_mon_vals no callback set 2024-09-13T11:59:08.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.651+0000 7f710143c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:59:08.668 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.651+0000 7f710143c640 10 monclient: dump: 2024-09-13T11:59:08.669 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:08.669 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.669 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.669 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.669 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:08.669 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.669 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:08.669 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:08.670 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:08.670 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.652+0000 7f71046a2640 10 monclient: shutdown 2024-09-13T11:59:08.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.652+0000 7f71046a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:08.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.652+0000 7f71046a2640 1 -- 172.21.15.100:0/3977390282 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70fc09f6d0 msgr2=0x7f70fc0a1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.652+0000 7f71046a2640 1 --2- 172.21.15.100:0/3977390282 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70fc09f6d0 0x7f70fc0a1ec0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f70ec0097a0 tx=0x7f70ec02a060 comp rx=0 tx=0).stop 2024-09-13T11:59:08.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.652+0000 7f71046a2640 1 -- 172.21.15.100:0/3977390282 shutdown_connections 2024-09-13T11:59:08.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.652+0000 7f71046a2640 1 --2- 172.21.15.100:0/3977390282 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70fc09f6d0 0x7f70fc0a1ec0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.652+0000 7f71046a2640 1 --2- 172.21.15.100:0/3977390282 >> v2:172.21.15.122:3300/0 conn(0x7f70fc0a15a0 0x7f70fc0a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.671 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.652+0000 7f71046a2640 1 -- 172.21.15.100:0/3977390282 >> 172.21.15.100:0/3977390282 conn(0x7f70fc0a6470 msgr2=0x7f70fc0a2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:08.671 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.652+0000 7f71046a2640 1 -- 172.21.15.100:0/3977390282 shutdown_connections 2024-09-13T11:59:08.671 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.652+0000 7f71046a2640 1 -- 172.21.15.100:0/3977390282 wait complete. 2024-09-13T11:59:08.671 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.652+0000 7f71046a2640 10 monclient: build_initial_monmap 2024-09-13T11:59:08.671 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.652+0000 7f71046a2640 10 monclient: monmap: 2024-09-13T11:59:08.671 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:08.671 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:08.671 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:08.671 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:08.671 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:08.671 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.671 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:08.672 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:08.672 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:08.672 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f71046a2640 1 Processor -- start 2024-09-13T11:59:08.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f71046a2640 1 -- start start 2024-09-13T11:59:08.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f71046a2640 10 monclient: init 2024-09-13T11:59:08.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f71046a2640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:08.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f71046a2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:59:08.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f71046a2640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70fc09f6d0 0x7f70fc06f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f71046a2640 10 monclient(hunting): picked mon.noname-a con 0x7f70fc09f6d0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:08.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f71046a2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70fc0a15a0 0x7f70fc06fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.672 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f71046a2640 10 monclient(hunting): picked mon.noname-b con 0x7f70fc0a15a0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:08.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f71046a2640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70fc1343d0 0x7f70fc070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f71046a2640 10 monclient(hunting): picked mon.noname-c con 0x7f70fc1343d0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:08.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f71046a2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:08.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f71046a2640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f70fc09d120 con 0x7f70fc09f6d0 2024-09-13T11:59:08.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f71046a2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:08.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f71046a2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f70fc09cfa0 con 0x7f70fc0a15a0 2024-09-13T11:59:08.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f71046a2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:08.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f71046a2640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f70fc134f70 con 0x7f70fc1343d0 2024-09-13T11:59:08.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f71046a2640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:08.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f71046a2640 10 monclient(hunting): authenticate will time out at 1129.247812s 2024-09-13T11:59:08.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f710243e640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70fc09f6d0 0x7f70fc06f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f7102c3f640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70fc1343d0 0x7f70fc070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.673 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f710243e640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70fc09f6d0 0x7f70fc06f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47292/0 (socket says 172.21.15.100:47292) 2024-09-13T11:59:08.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f710243e640 1 -- 172.21.15.100:0/508452857 learned_addr learned my addr 172.21.15.100:0/508452857 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:08.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f710243e640 10 monclient(hunting): get_auth_request con 0x7f70fc09f6d0 auth_method 0 2024-09-13T11:59:08.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f710243e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:08.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f710243e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f710243e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f7102c3f640 10 monclient(hunting): get_auth_request con 0x7f70fc1343d0 auth_method 0 2024-09-13T11:59:08.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f7102c3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:08.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f7102c3f640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f7102c3f640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f710243e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:08.674 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.653+0000 7f710243e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:08.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f710243e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:08.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f7102c3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:08.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f7102c3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:08.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f7102c3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:08.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f7101c3d640 1 --2- 172.21.15.100:0/508452857 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70fc0a15a0 0x7f70fc06fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f7101c3d640 10 monclient(hunting): get_auth_request con 0x7f70fc0a15a0 auth_method 0 2024-09-13T11:59:08.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f7101c3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:08.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f7101c3d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f7101c3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f7102c3f640 10 monclient(hunting): handle_auth_done global_id 24674 payload 824 2024-09-13T11:59:08.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f7102c3f640 1 -- 172.21.15.100:0/508452857 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70fc0a15a0 msgr2=0x7f70fc06fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f7102c3f640 1 --2- 172.21.15.100:0/508452857 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70fc0a15a0 0x7f70fc06fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f7102c3f640 1 -- 172.21.15.100:0/508452857 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70fc09f6d0 msgr2=0x7f70fc06f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f7102c3f640 1 --2- 172.21.15.100:0/508452857 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70fc09f6d0 0x7f70fc06f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f7102c3f640 10 monclient: _finish_hunting 0 2024-09-13T11:59:08.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f7102c3f640 1 monclient: found mon.noname-c 2024-09-13T11:59:08.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f7102c3f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:08.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f7102c3f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:59:08.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f7102c3f640 1 -- 172.21.15.100:0/508452857 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70fc127de0 con 0x7f70fc1343d0 2024-09-13T11:59:08.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f710243e640 1 --2- 172.21.15.100:0/508452857 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70fc09f6d0 0x7f70fc06f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:59:08.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f7102c3f640 1 --2- 172.21.15.100:0/508452857 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70fc1343d0 0x7f70fc070270 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f70f400c3f0 tx=0x7f70f400c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:08.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f70f2ffd640 1 -- 172.21.15.100:0/508452857 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f4017070 con 0x7f70fc1343d0 2024-09-13T11:59:08.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f70f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:08.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f70f2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:59:08.677 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f70f2ffd640 10 monclient: dump: 2024-09-13T11:59:08.677 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:08.677 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.677 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.677 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.677 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:08.677 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.677 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:08.677 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:08.678 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:08.678 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f70f2ffd640 10 monclient: _finish_auth 0 2024-09-13T11:59:08.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f70f2ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:59:08.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f70f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:08.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f70f2ffd640 1 -- 172.21.15.100:0/508452857 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70f4005ce0 con 0x7f70fc1343d0 2024-09-13T11:59:08.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f71046a2640 5 monclient: authenticate success, global_id 24674 2024-09-13T11:59:08.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f71046a2640 10 monclient: _renew_subs 2024-09-13T11:59:08.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f71046a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:59:08.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f71046a2640 1 -- 172.21.15.100:0/508452857 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70fc0a5340 con 0x7f70fc1343d0 2024-09-13T11:59:08.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f70f2ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:08.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f70f2ffd640 1 -- 172.21.15.100:0/508452857 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f40072d0 con 0x7f70fc1343d0 2024-09-13T11:59:08.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f70f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:08.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f70f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:59:08.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f70f2ffd640 10 monclient: dump: 2024-09-13T11:59:08.679 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:08.679 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.679 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.679 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.679 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:08.679 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.679 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:08.679 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:08.679 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:08.679 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f7100c3b640 4 set_mon_vals no callback set 2024-09-13T11:59:08.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f71046a2640 10 monclient: _renew_subs 2024-09-13T11:59:08.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f71046a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:59:08.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.654+0000 7f71046a2640 1 -- 172.21.15.100:0/508452857 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f70fc133e20 con 0x7f70fc1343d0 2024-09-13T11:59:08.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.655+0000 7f70f2ffd640 1 -- 172.21.15.100:0/508452857 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f70f401a070 con 0x7f70fc1343d0 2024-09-13T11:59:08.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.655+0000 7f71046a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:08.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.655+0000 7f71046a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:08.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.655+0000 7f71046a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:59:08.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.655+0000 7f71046a2640 1 -- 172.21.15.100:0/508452857 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70fc134860 con 0x7f70fc1343d0 2024-09-13T11:59:08.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.655+0000 7f70f2ffd640 1 --2- 172.21.15.100:0/508452857 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f70e003dfa0 0x7f70e0040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.655+0000 7f70f2ffd640 1 -- 172.21.15.100:0/508452857 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7f70f4051890 con 0x7f70fc1343d0 2024-09-13T11:59:08.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.656+0000 7f710243e640 1 --2- 172.21.15.100:0/508452857 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f70e003dfa0 0x7f70e0040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.656+0000 7f710243e640 10 monclient: get_auth_request con 0x7f70e003dfa0 auth_method 0 2024-09-13T11:59:08.680 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.656+0000 7f710243e640 1 --2- 172.21.15.100:0/508452857 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f70e003dfa0 0x7f70e0040460 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f70f8009a40 tx=0x7f70f800b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:08.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.660+0000 7f70f2ffd640 1 -- 172.21.15.100:0/508452857 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f70f401b3b0 con 0x7f70fc1343d0 2024-09-13T11:59:08.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.660+0000 7f70f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:08.681 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.660+0000 7f70f2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:59:08.836 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.833+0000 7f71046a2640 1 -- 172.21.15.100:0/508452857 --> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f70fc06ca60 con 0x7f70e003dfa0 2024-09-13T11:59:08.836 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.834+0000 7f70f2ffd640 1 -- 172.21.15.100:0/508452857 <== mgr.4114 v2:172.21.15.122:6824/441789830 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f70fc06ca60 con 0x7f70e003dfa0 2024-09-13T11:59:08.836 INFO:teuthology.orchestra.run.smithi100.stderr:instructed osd(s) 4 to deep-scrub 2024-09-13T11:59:08.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.836+0000 7f71046a2640 1 -- 172.21.15.100:0/508452857 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f70e003dfa0 msgr2=0x7f70e0040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.836+0000 7f71046a2640 1 --2- 172.21.15.100:0/508452857 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f70e003dfa0 0x7f70e0040460 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f70f8009a40 tx=0x7f70f800b040 comp rx=0 tx=0).stop 2024-09-13T11:59:08.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.836+0000 7f71046a2640 10 monclient: shutdown 2024-09-13T11:59:08.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.836+0000 7f71046a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:08.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.836+0000 7f71046a2640 1 -- 172.21.15.100:0/508452857 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70fc1343d0 msgr2=0x7f70fc070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.836+0000 7f71046a2640 1 --2- 172.21.15.100:0/508452857 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70fc1343d0 0x7f70fc070270 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f70f400c3f0 tx=0x7f70f400c8b0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.836+0000 7f71046a2640 1 -- 172.21.15.100:0/508452857 shutdown_connections 2024-09-13T11:59:08.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.836+0000 7f71046a2640 1 --2- 172.21.15.100:0/508452857 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f70e003dfa0 0x7f70e0040460 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.836+0000 7f71046a2640 1 --2- 172.21.15.100:0/508452857 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70fc1343d0 0x7f70fc070270 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.836+0000 7f71046a2640 1 --2- 172.21.15.100:0/508452857 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70fc0a15a0 0x7f70fc06fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.836+0000 7f71046a2640 1 --2- 172.21.15.100:0/508452857 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70fc09f6d0 0x7f70fc06f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.836+0000 7f71046a2640 1 -- 172.21.15.100:0/508452857 >> 172.21.15.100:0/508452857 conn(0x7f70fc0a6470 msgr2=0x7f70fc09bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:08.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.836+0000 7f71046a2640 1 -- 172.21.15.100:0/508452857 shutdown_connections 2024-09-13T11:59:08.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.836+0000 7f71046a2640 1 -- 172.21.15.100:0/508452857 wait complete. 2024-09-13T11:59:08.849 INFO:tasks.ceph:Scrubbing osd.5 2024-09-13T11:59:08.849 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:59:08.951 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.949+0000 7f816b7f8640 10 monclient: get_monmap_and_config 2024-09-13T11:59:08.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.949+0000 7f816b7f8640 10 monclient: build_initial_monmap 2024-09-13T11:59:08.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.949+0000 7f816b7f8640 10 monclient: monmap: 2024-09-13T11:59:08.952 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:08.952 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.952 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:08.951437+0000 2024-09-13T11:59:08.952 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:08.951437+0000 2024-09-13T11:59:08.952 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:08.952 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.952 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:08.952 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:08.952 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:08.953 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:08.953 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:08.953 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.950+0000 7f816b7f8640 1 Processor -- start 2024-09-13T11:59:08.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.950+0000 7f816b7f8640 1 -- start start 2024-09-13T11:59:08.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.950+0000 7f816b7f8640 10 monclient: init 2024-09-13T11:59:08.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.950+0000 7f816b7f8640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:08.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.950+0000 7f816b7f8640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-13T11:59:08.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.950+0000 7f816b7f8640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81640a3410 0x7f81640a2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.950+0000 7f816b7f8640 10 monclient(hunting): picked mon.noname-b con 0x7f81640a3410 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:08.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.950+0000 7f816b7f8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f81640a37f0 addr v1:172.21.15.100:6789/0 2024-09-13T11:59:08.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.950+0000 7f816b7f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f81640a3bf0 addr v1:172.21.15.122:6789/0 2024-09-13T11:59:08.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.950+0000 7f816b7f8640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81640a5700 con 0x7f81640a37f0 2024-09-13T11:59:08.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.950+0000 7f816b7f8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8164070530 con 0x7f81640a3bf0 2024-09-13T11:59:08.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.950+0000 7f816b7f8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:08.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.950+0000 7f816b7f8640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f81640a1ce0 con 0x7f81640a3410 2024-09-13T11:59:08.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.950+0000 7f816b7f8640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:08.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.950+0000 7f816b7f8640 10 monclient(hunting): authenticate will time out at 1129.544907s 2024-09-13T11:59:08.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.950+0000 7f8169594640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81640a3410 0x7f81640a2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.951+0000 7f8168d93640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f81640a37f0 0x7f81640a48e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:57146/0 (socket says 172.21.15.100:57146) 2024-09-13T11:59:08.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.951+0000 7f8168d93640 1 -- 172.21.15.100:0/3086971972 learned_addr learned my addr 172.21.15.100:0/3086971972 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:59:08.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.951+0000 7f8169594640 10 monclient(hunting): get_auth_request con 0x7f81640a3410 auth_method 0 2024-09-13T11:59:08.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.951+0000 7f8169594640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:08.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.951+0000 7f8169594640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.951+0000 7f8169594640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.951+0000 7f8169594640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:08.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.951+0000 7f8169594640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:08.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.951+0000 7f8169594640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:08.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.951+0000 7f815bfff640 1 -- 172.21.15.100:0/3086971972 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 234510066 0 0) 0x7f81640a5700 con 0x7f81640a37f0 2024-09-13T11:59:08.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.951+0000 7f815bfff640 10 monclient(hunting): _negotiate 2024-09-13T11:59:08.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.951+0000 7f815bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.951+0000 7f815bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.951+0000 7f815bfff640 10 monclient(hunting): my global_id is 15588 2024-09-13T11:59:08.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.951+0000 7f815bfff640 1 -- 172.21.15.100:0/3086971972 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8150003520 con 0x7f81640a37f0 2024-09-13T11:59:08.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.951+0000 7f8169594640 10 monclient(hunting): handle_auth_done global_id 24677 payload 274 2024-09-13T11:59:08.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.951+0000 7f8169594640 1 -- 172.21.15.100:0/3086971972 >> v1:172.21.15.122:6789/0 conn(0x7f81640a3bf0 legacy=0x7f81640a4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:08.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.951+0000 7f8169594640 1 -- 172.21.15.100:0/3086971972 >> v1:172.21.15.100:6789/0 conn(0x7f81640a37f0 legacy=0x7f81640a48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.951+0000 7f8169594640 10 monclient: _finish_hunting 0 2024-09-13T11:59:08.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.951+0000 7f8169594640 1 monclient: found mon.noname-b 2024-09-13T11:59:08.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.951+0000 7f8169594640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:08.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.951+0000 7f8169594640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:59:08.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.951+0000 7f8169594640 1 -- 172.21.15.100:0/3086971972 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81640a43c0 con 0x7f81640a3410 2024-09-13T11:59:08.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f8169594640 1 --2- 172.21.15.100:0/3086971972 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81640a3410 0x7f81640a2960 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f81600097d0 tx=0x7f816002a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc61ab23eed1b2d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:08.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f815bfff640 1 -- 172.21.15.100:0/3086971972 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8160037070 con 0x7f81640a3410 2024-09-13T11:59:08.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f815bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:08.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f815bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:59:08.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f815bfff640 10 monclient: dump: 2024-09-13T11:59:08.965 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:08.965 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.965 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.965 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.966 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:08.966 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.966 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:08.966 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:08.966 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:08.966 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f815bfff640 10 monclient: _finish_auth 0 2024-09-13T11:59:08.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f815bfff640 10 monclient: _check_auth_tickets 2024-09-13T11:59:08.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f815bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:08.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f816b7f8640 5 monclient: authenticate success, global_id 24677 2024-09-13T11:59:08.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f815bfff640 1 -- 172.21.15.100:0/3086971972 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f816002ad90 con 0x7f81640a3410 2024-09-13T11:59:08.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f816b7f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:08.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f815bfff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:08.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f816b7f8640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:08.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f816b7f8640 4 set_mon_vals no callback set 2024-09-13T11:59:08.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f815bfff640 1 -- 172.21.15.100:0/3086971972 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8160032660 con 0x7f81640a3410 2024-09-13T11:59:08.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f815bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:08.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f815bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:59:08.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f815bfff640 10 monclient: dump: 2024-09-13T11:59:08.967 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:08.967 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.968 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.968 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.968 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:08.968 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.968 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:08.968 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:08.968 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:08.968 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f816b7f8640 10 monclient: shutdown 2024-09-13T11:59:08.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f816b7f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:08.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f816b7f8640 1 -- 172.21.15.100:0/3086971972 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81640a3410 msgr2=0x7f81640a2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f816b7f8640 1 --2- 172.21.15.100:0/3086971972 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81640a3410 0x7f81640a2960 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f81600097d0 tx=0x7f816002a040 comp rx=0 tx=0).stop 2024-09-13T11:59:08.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f816b7f8640 1 -- 172.21.15.100:0/3086971972 shutdown_connections 2024-09-13T11:59:08.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f816b7f8640 1 --2- 172.21.15.100:0/3086971972 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81640a3410 0x7f81640a2960 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f816b7f8640 1 -- 172.21.15.100:0/3086971972 >> 172.21.15.100:0/3086971972 conn(0x7f81640aaae0 msgr2=0x7f81640a69e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:08.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f816b7f8640 1 -- 172.21.15.100:0/3086971972 shutdown_connections 2024-09-13T11:59:08.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.952+0000 7f816b7f8640 1 -- 172.21.15.100:0/3086971972 wait complete. 2024-09-13T11:59:08.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.953+0000 7f816b7f8640 10 monclient: build_initial_monmap 2024-09-13T11:59:08.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.953+0000 7f816b7f8640 10 monclient: monmap: 2024-09-13T11:59:08.969 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:08.969 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:08.969 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:08.969 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:08.970 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:08.970 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.970 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:08.970 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:08.970 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:08.970 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.953+0000 7f816b7f8640 1 Processor -- start 2024-09-13T11:59:08.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.953+0000 7f816b7f8640 1 -- start start 2024-09-13T11:59:08.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.953+0000 7f816b7f8640 10 monclient: init 2024-09-13T11:59:08.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f816b7f8640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:08.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f816b7f8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:59:08.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f816b7f8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81640a3410 0x7f8164127ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f816b7f8640 10 monclient(hunting): picked mon.noname-b con 0x7f81640a3410 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:08.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f816b7f8640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81640a37f0 0x7f816413f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f816b7f8640 10 monclient(hunting): picked mon.noname-c con 0x7f81640a37f0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:08.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f816b7f8640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81640a3bf0 0x7f8164125500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f816b7f8640 10 monclient(hunting): picked mon.noname-a con 0x7f81640a3bf0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:08.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f816b7f8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:08.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f816b7f8640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f816406fcd0 con 0x7f81640a3bf0 2024-09-13T11:59:08.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f816b7f8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:08.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f816b7f8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f81640a1690 con 0x7f81640a3410 2024-09-13T11:59:08.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f816b7f8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:08.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f816b7f8640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f816406fe50 con 0x7f81640a37f0 2024-09-13T11:59:08.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f816b7f8640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:08.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f816b7f8640 10 monclient(hunting): authenticate will time out at 1129.548202s 2024-09-13T11:59:08.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8168d93640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81640a37f0 0x7f816413f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8169d95640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81640a3bf0 0x7f8164125500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8168d93640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81640a37f0 0x7f816413f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:52154/0 (socket says 172.21.15.100:52154) 2024-09-13T11:59:08.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8168d93640 1 -- 172.21.15.100:0/3098460759 learned_addr learned my addr 172.21.15.100:0/3098460759 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:08.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8168d93640 10 monclient(hunting): get_auth_request con 0x7f81640a37f0 auth_method 0 2024-09-13T11:59:08.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8168d93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:08.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8168d93640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8168d93640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8169d95640 10 monclient(hunting): get_auth_request con 0x7f81640a3bf0 auth_method 0 2024-09-13T11:59:08.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8169d95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:08.972 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8169d95640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:08.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8169d95640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:08.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8169d95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:08.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8169d95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:08.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8169d95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:08.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8168d93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:08.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8168d93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:08.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8168d93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:08.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8169d95640 10 monclient(hunting): handle_auth_done global_id 15591 payload 824 2024-09-13T11:59:08.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8169594640 1 --2- 172.21.15.100:0/3098460759 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81640a3410 0x7f8164127ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8169d95640 1 -- 172.21.15.100:0/3098460759 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81640a37f0 msgr2=0x7f816413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8169d95640 1 --2- 172.21.15.100:0/3098460759 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81640a37f0 0x7f816413f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8169d95640 1 -- 172.21.15.100:0/3098460759 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81640a3410 msgr2=0x7f8164127ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8169d95640 1 --2- 172.21.15.100:0/3098460759 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81640a3410 0x7f8164127ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8169d95640 10 monclient: _finish_hunting 0 2024-09-13T11:59:08.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8169d95640 1 monclient: found mon.noname-a 2024-09-13T11:59:08.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8169d95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:08.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8169d95640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:08.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8169d95640 1 -- 172.21.15.100:0/3098460759 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8164129000 con 0x7f81640a3bf0 2024-09-13T11:59:08.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8169d95640 1 --2- 172.21.15.100:0/3098460759 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81640a3bf0 0x7f8164125500 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f815c00c780 tx=0x7f815c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:08.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8159ffb640 1 -- 172.21.15.100:0/3098460759 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815c016020 con 0x7f81640a3bf0 2024-09-13T11:59:08.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8159ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:08.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8159ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:08.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8159ffb640 10 monclient: dump: 2024-09-13T11:59:08.974 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:08.974 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.974 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.975 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.975 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:08.975 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.975 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:08.975 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:08.975 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:08.975 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.975 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8159ffb640 10 monclient: _finish_auth 0 2024-09-13T11:59:08.975 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8159ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:59:08.975 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8159ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:08.975 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f816b7f8640 5 monclient: authenticate success, global_id 15591 2024-09-13T11:59:08.975 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8159ffb640 1 -- 172.21.15.100:0/3098460759 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f815c00aa50 con 0x7f81640a3bf0 2024-09-13T11:59:08.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8159ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:08.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8159ffb640 1 -- 172.21.15.100:0/3098460759 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815c00ae40 con 0x7f81640a3bf0 2024-09-13T11:59:08.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f816b7f8640 10 monclient: _renew_subs 2024-09-13T11:59:08.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f816b7f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:08.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f816b7f8640 1 -- 172.21.15.100:0/3098460759 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81640a10d0 con 0x7f81640a3bf0 2024-09-13T11:59:08.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8159ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:08.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8159ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:08.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f8159ffb640 10 monclient: dump: 2024-09-13T11:59:08.976 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:08.976 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:08.976 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.976 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:08.977 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:08.977 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:08.977 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:08.977 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:08.977 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:08.977 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:08.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.954+0000 7f815b7fe640 4 set_mon_vals no callback set 2024-09-13T11:59:08.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.955+0000 7f816b7f8640 10 monclient: _renew_subs 2024-09-13T11:59:08.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.955+0000 7f816b7f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:08.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.955+0000 7f816b7f8640 1 -- 172.21.15.100:0/3098460759 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8164128d20 con 0x7f81640a3bf0 2024-09-13T11:59:08.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.955+0000 7f8159ffb640 1 -- 172.21.15.100:0/3098460759 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f815c019020 con 0x7f81640a3bf0 2024-09-13T11:59:08.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.955+0000 7f816b7f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:08.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.955+0000 7f816b7f8640 1 -- 172.21.15.100:0/3098460759 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f81640a2960 con 0x7f81640a3bf0 2024-09-13T11:59:08.977 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.956+0000 7f8159ffb640 1 --2- 172.21.15.100:0/3098460759 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f814803dfd0 0x7f8148040490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.956+0000 7f8159ffb640 1 -- 172.21.15.100:0/3098460759 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7f815c0520a0 con 0x7f81640a3bf0 2024-09-13T11:59:08.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.956+0000 7f8159ffb640 1 --2- 172.21.15.100:0/3098460759 >> [v2:172.21.15.122:6800/904725860,v1:172.21.15.122:6801/904725860] conn(0x7f8148043e40 0x7f8148046260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:08.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.956+0000 7f8159ffb640 1 -- 172.21.15.100:0/3098460759 --> [v2:172.21.15.122:6800/904725860,v1:172.21.15.122:6801/904725860] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8148046910 con 0x7f8148043e40 2024-09-13T11:59:08.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.956+0000 7f8159ffb640 1 -- 172.21.15.100:0/3098460759 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_get_version_reply(handle=1 version=313) ==== 24+0+0 (secure 0 0 0) 0x7f815c059050 con 0x7f81640a3bf0 2024-09-13T11:59:08.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.956+0000 7f8159ffb640 10 monclient: handle_get_version_reply finishing 1 version 313 2024-09-13T11:59:08.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.956+0000 7f8169594640 1 --2- 172.21.15.100:0/3098460759 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f814803dfd0 0x7f8148040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.956+0000 7f8169594640 10 monclient: get_auth_request con 0x7f814803dfd0 auth_method 0 2024-09-13T11:59:08.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.956+0000 7f8168d93640 1 --2- 172.21.15.100:0/3098460759 >> [v2:172.21.15.122:6800/904725860,v1:172.21.15.122:6801/904725860] conn(0x7f8148043e40 0x7f8148046260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:08.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.956+0000 7f8168d93640 10 monclient: get_auth_request con 0x7f8148043e40 auth_method 0 2024-09-13T11:59:08.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.956+0000 7f8169594640 1 --2- 172.21.15.100:0/3098460759 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f814803dfd0 0x7f8148040490 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8160034000 tx=0x7f816002e000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:08.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.956+0000 7f8168d93640 1 --2- 172.21.15.100:0/3098460759 >> [v2:172.21.15.122:6800/904725860,v1:172.21.15.122:6801/904725860] conn(0x7f8148043e40 0x7f8148046260 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:08.978 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.957+0000 7f8159ffb640 1 -- 172.21.15.100:0/3098460759 <== osd.5 v2:172.21.15.122:6800/904725860 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8148046910 con 0x7f8148043e40 2024-09-13T11:59:08.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.969+0000 7f816b7f8640 1 -- 172.21.15.100:0/3098460759 --> [v2:172.21.15.122:6800/904725860,v1:172.21.15.122:6801/904725860] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f816409fc10 con 0x7f8148043e40 2024-09-13T11:59:08.979 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.970+0000 7f8159ffb640 1 -- 172.21.15.100:0/3098460759 <== osd.5 v2:172.21.15.122:6800/904725860 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f816409fc10 con 0x7f8148043e40 2024-09-13T11:59:08.979 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T11:59:08.979 INFO:teuthology.orchestra.run.smithi100.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:59:08.980 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T11:59:08.980 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.970+0000 7f816b7f8640 1 -- 172.21.15.100:0/3098460759 >> [v2:172.21.15.122:6800/904725860,v1:172.21.15.122:6801/904725860] conn(0x7f8148043e40 msgr2=0x7f8148046260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.970+0000 7f816b7f8640 1 --2- 172.21.15.100:0/3098460759 >> [v2:172.21.15.122:6800/904725860,v1:172.21.15.122:6801/904725860] conn(0x7f8148043e40 0x7f8148046260 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:08.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.970+0000 7f816b7f8640 1 -- 172.21.15.100:0/3098460759 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f814803dfd0 msgr2=0x7f8148040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.970+0000 7f816b7f8640 1 --2- 172.21.15.100:0/3098460759 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f814803dfd0 0x7f8148040490 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8160034000 tx=0x7f816002e000 comp rx=0 tx=0).stop 2024-09-13T11:59:08.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.970+0000 7f816b7f8640 10 monclient: shutdown 2024-09-13T11:59:08.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.970+0000 7f816b7f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:08.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.970+0000 7f816b7f8640 1 -- 172.21.15.100:0/3098460759 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81640a3bf0 msgr2=0x7f8164125500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:08.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.970+0000 7f816b7f8640 1 --2- 172.21.15.100:0/3098460759 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81640a3bf0 0x7f8164125500 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f815c00c780 tx=0x7f815c00cc40 comp rx=0 tx=0).stop 2024-09-13T11:59:08.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.970+0000 7f8169d95640 1 -- 172.21.15.100:0/3098460759 reap_dead start 2024-09-13T11:59:08.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.970+0000 7f816b7f8640 1 -- 172.21.15.100:0/3098460759 shutdown_connections 2024-09-13T11:59:08.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.970+0000 7f816b7f8640 1 -- 172.21.15.100:0/3098460759 >> 172.21.15.100:0/3098460759 conn(0x7f81640aaae0 msgr2=0x7f81640a8600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:08.981 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.970+0000 7f816b7f8640 1 -- 172.21.15.100:0/3098460759 shutdown_connections 2024-09-13T11:59:08.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:08.970+0000 7f816b7f8640 1 -- 172.21.15.100:0/3098460759 wait complete. 2024-09-13T11:59:08.982 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-13T11:59:09.083 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.080+0000 7f108ae40640 10 monclient: get_monmap_and_config 2024-09-13T11:59:09.083 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.080+0000 7f108ae40640 10 monclient: build_initial_monmap 2024-09-13T11:59:09.083 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.080+0000 7f108ae40640 10 monclient: monmap: 2024-09-13T11:59:09.083 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:09.083 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:09.083 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:09.082785+0000 2024-09-13T11:59:09.084 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:09.082785+0000 2024-09-13T11:59:09.084 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:09.084 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:09.084 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:09.084 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:09.084 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:09.084 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:09.084 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:09.084 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:09.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.081+0000 7f108ae40640 1 Processor -- start 2024-09-13T11:59:09.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.081+0000 7f108ae40640 1 -- start start 2024-09-13T11:59:09.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.081+0000 7f108ae40640 10 monclient: init 2024-09-13T11:59:09.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f108ae40640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:09.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f108ae40640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-13T11:59:09.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f108ae40640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f10840a4870 0x7f10840a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:09.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f108ae40640 10 monclient(hunting): picked mon.noname-a con 0x7f10840a4870 addr v2:172.21.15.100:3300/0 2024-09-13T11:59:09.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f108ae40640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f10840a5190 0x7f1084137630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:09.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f108ae40640 10 monclient(hunting): picked mon.noname-c con 0x7f10840a5190 addr v2:172.21.15.122:3300/0 2024-09-13T11:59:09.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f108ae40640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1084137b70 addr v1:172.21.15.122:6789/0 2024-09-13T11:59:09.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f108ae40640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10840a1df0 con 0x7f1084137b70 2024-09-13T11:59:09.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f108ae40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:09.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f108ae40640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7f10840a5960 con 0x7f10840a4870 2024-09-13T11:59:09.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f108ae40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:09.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f108ae40640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f10840a0ed0 con 0x7f10840a5190 2024-09-13T11:59:09.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f108ae40640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:09.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f108ae40640 10 monclient(hunting): authenticate will time out at 1129.676305s 2024-09-13T11:59:09.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f1088bdc640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f10840a4870 0x7f10840a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:09.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f1088bdc640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f10840a4870 0x7f10840a4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47312/0 (socket says 172.21.15.100:47312) 2024-09-13T11:59:09.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f1088bdc640 1 -- 172.21.15.100:0/714372472 learned_addr learned my addr 172.21.15.100:0/714372472 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:09.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f1088bdc640 10 monclient(hunting): get_auth_request con 0x7f10840a4870 auth_method 0 2024-09-13T11:59:09.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f1088bdc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:09.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f1088bdc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:09.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f1088bdc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:09.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f1088bdc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:09.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f1088bdc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:09.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f1088bdc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:09.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f107bfff640 1 --2- 172.21.15.100:0/714372472 >> v2:172.21.15.122:3300/0 conn(0x7f10840a5190 0x7f1084137630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:09.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f107bfff640 10 monclient(hunting): get_auth_request con 0x7f10840a5190 auth_method 0 2024-09-13T11:59:09.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f107bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:09.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f107bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:09.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f107bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:09.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f1088bdc640 10 monclient(hunting): handle_auth_done global_id 15594 payload 274 2024-09-13T11:59:09.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f1088bdc640 1 -- 172.21.15.100:0/714372472 >> v2:172.21.15.122:3300/0 conn(0x7f10840a5190 msgr2=0x7f1084137630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:09.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f1088bdc640 1 --2- 172.21.15.100:0/714372472 >> v2:172.21.15.122:3300/0 conn(0x7f10840a5190 0x7f1084137630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:09.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f1088bdc640 1 -- 172.21.15.100:0/714372472 >> v1:172.21.15.122:6789/0 conn(0x7f1084137b70 legacy=0x7f108413ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:09.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f1088bdc640 10 monclient: _finish_hunting 0 2024-09-13T11:59:09.096 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f1088bdc640 1 monclient: found mon.noname-a 2024-09-13T11:59:09.096 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f1088bdc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:09.096 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f1088bdc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:09.096 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.082+0000 7f1088bdc640 1 -- 172.21.15.100:0/714372472 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1084070410 con 0x7f10840a4870 2024-09-13T11:59:09.096 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.083+0000 7f1088bdc640 1 --2- 172.21.15.100:0/714372472 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f10840a4870 0x7f10840a4c50 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f1084077410 tx=0x7f108002a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c447dd6d8331c73 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:09.096 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.083+0000 7f107b7fe640 1 -- 172.21.15.100:0/714372472 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1080037070 con 0x7f10840a4870 2024-09-13T11:59:09.096 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.083+0000 7f107b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:09.096 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.083+0000 7f107b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:09.096 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.083+0000 7f107b7fe640 10 monclient: dump: 2024-09-13T11:59:09.096 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:09.096 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:09.096 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:09.096 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:09.097 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:09.097 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:09.097 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:09.097 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:09.097 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:09.097 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:09.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.083+0000 7f107b7fe640 10 monclient: _finish_auth 0 2024-09-13T11:59:09.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.083+0000 7f107b7fe640 10 monclient: _check_auth_tickets 2024-09-13T11:59:09.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.083+0000 7f107b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:09.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.083+0000 7f107b7fe640 1 -- 172.21.15.100:0/714372472 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f108002ae00 con 0x7f10840a4870 2024-09-13T11:59:09.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.083+0000 7f108ae40640 5 monclient: authenticate success, global_id 15594 2024-09-13T11:59:09.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.083+0000 7f108ae40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:09.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.083+0000 7f107b7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:09.098 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.083+0000 7f108ae40640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:09.098 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.083+0000 7f107b7fe640 1 -- 172.21.15.100:0/714372472 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1080032660 con 0x7f10840a4870 2024-09-13T11:59:09.098 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.083+0000 7f107b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:09.098 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.083+0000 7f108ae40640 4 set_mon_vals no callback set 2024-09-13T11:59:09.098 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.083+0000 7f107b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:09.098 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.083+0000 7f107b7fe640 10 monclient: dump: 2024-09-13T11:59:09.098 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:09.098 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:09.098 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:09.098 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:09.098 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:09.098 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:09.098 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:09.098 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:09.099 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:09.099 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:09.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.084+0000 7f108ae40640 10 monclient: shutdown 2024-09-13T11:59:09.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.084+0000 7f108ae40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:09.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.084+0000 7f108ae40640 1 -- 172.21.15.100:0/714372472 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f10840a4870 msgr2=0x7f10840a4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:09.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.084+0000 7f108ae40640 1 --2- 172.21.15.100:0/714372472 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f10840a4870 0x7f10840a4c50 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f1084077410 tx=0x7f108002a0b0 comp rx=0 tx=0).stop 2024-09-13T11:59:09.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.084+0000 7f108ae40640 1 -- 172.21.15.100:0/714372472 shutdown_connections 2024-09-13T11:59:09.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.084+0000 7f108ae40640 1 --2- 172.21.15.100:0/714372472 >> v2:172.21.15.122:3300/0 conn(0x7f10840a5190 0x7f1084137630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:09.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.084+0000 7f108ae40640 1 --2- 172.21.15.100:0/714372472 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f10840a4870 0x7f10840a4c50 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:09.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.084+0000 7f108ae40640 1 -- 172.21.15.100:0/714372472 >> 172.21.15.100:0/714372472 conn(0x7f10840a6970 msgr2=0x7f10840a6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:09.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.084+0000 7f108ae40640 1 -- 172.21.15.100:0/714372472 shutdown_connections 2024-09-13T11:59:09.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.084+0000 7f108ae40640 1 -- 172.21.15.100:0/714372472 wait complete. 2024-09-13T11:59:09.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.084+0000 7f108ae40640 10 monclient: build_initial_monmap 2024-09-13T11:59:09.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.084+0000 7f108ae40640 10 monclient: monmap: 2024-09-13T11:59:09.100 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:09.100 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:09.100 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:09.100 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:09.100 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:09.100 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:09.100 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:09.100 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:09.100 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:09.100 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:09.100 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f108ae40640 1 Processor -- start 2024-09-13T11:59:09.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f108ae40640 1 -- start start 2024-09-13T11:59:09.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f108ae40640 10 monclient: init 2024-09-13T11:59:09.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f108ae40640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:09.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f108ae40640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:59:09.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f108ae40640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f10840a4870 0x7f10841ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:09.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f108ae40640 10 monclient(hunting): picked mon.noname-a con 0x7f10840a4870 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:09.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f108ae40640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10840a5190 0x7f10841cd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:09.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f108ae40640 10 monclient(hunting): picked mon.noname-c con 0x7f10840a5190 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:09.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f108ae40640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1084137b70 0x7f1084123fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:09.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f108ae40640 10 monclient(hunting): picked mon.noname-b con 0x7f1084137b70 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:09.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f108ae40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:09.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f108ae40640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f108406fc60 con 0x7f10840a4870 2024-09-13T11:59:09.101 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f108ae40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:09.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f108ae40640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f10840a55a0 con 0x7f1084137b70 2024-09-13T11:59:09.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f108ae40640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:09.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f108ae40640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f108406fde0 con 0x7f10840a5190 2024-09-13T11:59:09.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f108ae40640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:09.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f108ae40640 10 monclient(hunting): authenticate will time out at 1129.679681s 2024-09-13T11:59:09.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f1088bdc640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f10840a4870 0x7f10841ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:09.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f1088bdc640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f10840a4870 0x7f10841ccad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47326/0 (socket says 172.21.15.100:47326) 2024-09-13T11:59:09.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f1088bdc640 1 -- 172.21.15.100:0/107273612 learned_addr learned my addr 172.21.15.100:0/107273612 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:09.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f1088bdc640 10 monclient(hunting): get_auth_request con 0x7f10840a4870 auth_method 0 2024-09-13T11:59:09.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f1088bdc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:09.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f1088bdc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:09.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f1088bdc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:09.102 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f107bfff640 1 --2- 172.21.15.100:0/107273612 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10840a5190 0x7f10841cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:09.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f107bfff640 10 monclient(hunting): get_auth_request con 0x7f10840a5190 auth_method 0 2024-09-13T11:59:09.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f107bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:09.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f107bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:09.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f107bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:09.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f1088bdc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:09.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f1088bdc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:09.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f1088bdc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:09.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f10893dd640 1 --2- 172.21.15.100:0/107273612 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1084137b70 0x7f1084123fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:09.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f107bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:09.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f107bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:09.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f107bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:09.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f10893dd640 10 monclient(hunting): get_auth_request con 0x7f1084137b70 auth_method 0 2024-09-13T11:59:09.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f10893dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:09.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f10893dd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:09.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.085+0000 7f10893dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:09.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f10893dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:09.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f10893dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:09.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f10893dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:09.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f1088bdc640 10 monclient(hunting): handle_auth_done global_id 15597 payload 824 2024-09-13T11:59:09.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f1088bdc640 1 -- 172.21.15.100:0/107273612 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10840a5190 msgr2=0x7f10841cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:09.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f1088bdc640 1 --2- 172.21.15.100:0/107273612 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10840a5190 0x7f10841cd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:09.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f1088bdc640 1 -- 172.21.15.100:0/107273612 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1084137b70 msgr2=0x7f1084123fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:09.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f1088bdc640 1 --2- 172.21.15.100:0/107273612 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1084137b70 0x7f1084123fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:09.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f1088bdc640 10 monclient: _finish_hunting 0 2024-09-13T11:59:09.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f1088bdc640 1 monclient: found mon.noname-a 2024-09-13T11:59:09.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f1088bdc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:09.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f1088bdc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:09.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f1088bdc640 1 -- 172.21.15.100:0/107273612 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10840a10d0 con 0x7f10840a4870 2024-09-13T11:59:09.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f1088bdc640 1 --2- 172.21.15.100:0/107273612 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f10840a4870 0x7f10841ccad0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1080000c00 tx=0x7f10800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:09.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f10797fa640 1 -- 172.21.15.100:0/107273612 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1080041070 con 0x7f10840a4870 2024-09-13T11:59:09.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f10797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:09.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f10797fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:09.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f10797fa640 10 monclient: dump: 2024-09-13T11:59:09.105 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:09.105 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:09.105 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:09.106 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:09.106 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:09.106 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:09.106 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:09.106 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:09.106 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:09.106 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:09.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f10797fa640 10 monclient: _finish_auth 0 2024-09-13T11:59:09.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f10797fa640 10 monclient: _check_auth_tickets 2024-09-13T11:59:09.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f10797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:09.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f10797fa640 1 -- 172.21.15.100:0/107273612 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f108003c460 con 0x7f10840a4870 2024-09-13T11:59:09.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f108ae40640 5 monclient: authenticate success, global_id 15597 2024-09-13T11:59:09.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f108ae40640 10 monclient: _renew_subs 2024-09-13T11:59:09.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f108ae40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:09.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f108ae40640 1 -- 172.21.15.100:0/107273612 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10840a05b0 con 0x7f10840a4870 2024-09-13T11:59:09.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f10797fa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:09.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f10797fa640 1 -- 172.21.15.100:0/107273612 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1080037040 con 0x7f10840a4870 2024-09-13T11:59:09.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f10797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:09.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f10797fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:09.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f10797fa640 10 monclient: dump: 2024-09-13T11:59:09.107 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:09.107 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:09.107 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:09.107 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:09.107 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:09.107 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:09.108 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:09.108 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:09.108 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:09.108 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:09.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f108ae40640 10 monclient: _renew_subs 2024-09-13T11:59:09.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f107affd640 4 set_mon_vals no callback set 2024-09-13T11:59:09.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f108ae40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:09.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.086+0000 7f108ae40640 1 -- 172.21.15.100:0/107273612 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1084127dd0 con 0x7f10840a4870 2024-09-13T11:59:09.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.087+0000 7f10797fa640 1 -- 172.21.15.100:0/107273612 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f108003a070 con 0x7f10840a4870 2024-09-13T11:59:09.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.087+0000 7f108ae40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:09.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.087+0000 7f108ae40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:09.108 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.087+0000 7f108ae40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:09.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.087+0000 7f108ae40640 1 -- 172.21.15.100:0/107273612 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f108406ffb0 con 0x7f10840a4870 2024-09-13T11:59:09.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.087+0000 7f10797fa640 1 --2- 172.21.15.100:0/107273612 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f107003dfa0 0x7f1070040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:09.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.087+0000 7f10797fa640 1 -- 172.21.15.100:0/107273612 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7f1080072870 con 0x7f10840a4870 2024-09-13T11:59:09.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.087+0000 7f107bfff640 1 --2- 172.21.15.100:0/107273612 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f107003dfa0 0x7f1070040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:09.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.087+0000 7f107bfff640 10 monclient: get_auth_request con 0x7f107003dfa0 auth_method 0 2024-09-13T11:59:09.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.088+0000 7f107bfff640 1 --2- 172.21.15.100:0/107273612 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f107003dfa0 0x7f1070040460 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f10840a4500 tx=0x7f1074008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:09.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.092+0000 7f10797fa640 1 -- 172.21.15.100:0/107273612 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f108406ffb0 con 0x7f10840a4870 2024-09-13T11:59:09.109 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.092+0000 7f10797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:09.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.092+0000 7f10797fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:59:09.268 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.266+0000 7f108ae40640 1 -- 172.21.15.100:0/107273612 --> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f10840a2860 con 0x7f107003dfa0 2024-09-13T11:59:09.269 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.266+0000 7f10797fa640 1 -- 172.21.15.100:0/107273612 <== mgr.4114 v2:172.21.15.122:6824/441789830 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f10840a2860 con 0x7f107003dfa0 2024-09-13T11:59:09.269 INFO:teuthology.orchestra.run.smithi100.stderr:instructed osd(s) 5 to deep-scrub 2024-09-13T11:59:09.271 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.268+0000 7f108ae40640 1 -- 172.21.15.100:0/107273612 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f107003dfa0 msgr2=0x7f1070040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:09.271 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.268+0000 7f108ae40640 1 --2- 172.21.15.100:0/107273612 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f107003dfa0 0x7f1070040460 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f10840a4500 tx=0x7f1074008040 comp rx=0 tx=0).stop 2024-09-13T11:59:09.271 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.268+0000 7f108ae40640 10 monclient: shutdown 2024-09-13T11:59:09.271 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.268+0000 7f108ae40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:09.271 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.268+0000 7f108ae40640 1 -- 172.21.15.100:0/107273612 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f10840a4870 msgr2=0x7f10841ccad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:09.271 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.268+0000 7f108ae40640 1 --2- 172.21.15.100:0/107273612 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f10840a4870 0x7f10841ccad0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1080000c00 tx=0x7f10800046c0 comp rx=0 tx=0).stop 2024-09-13T11:59:09.271 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.269+0000 7f108ae40640 1 -- 172.21.15.100:0/107273612 shutdown_connections 2024-09-13T11:59:09.271 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.269+0000 7f108ae40640 1 --2- 172.21.15.100:0/107273612 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f107003dfa0 0x7f1070040460 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:09.271 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.269+0000 7f108ae40640 1 --2- 172.21.15.100:0/107273612 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1084137b70 0x7f1084123fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:09.272 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.269+0000 7f108ae40640 1 --2- 172.21.15.100:0/107273612 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f10840a5190 0x7f10841cd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:09.272 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.269+0000 7f108ae40640 1 --2- 172.21.15.100:0/107273612 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f10840a4870 0x7f10841ccad0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:09.272 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.269+0000 7f108ae40640 1 -- 172.21.15.100:0/107273612 >> 172.21.15.100:0/107273612 conn(0x7f10840a6970 msgr2=0x7f10840a2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:09.272 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.269+0000 7f108ae40640 1 -- 172.21.15.100:0/107273612 shutdown_connections 2024-09-13T11:59:09.272 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.269+0000 7f108ae40640 1 -- 172.21.15.100:0/107273612 wait complete. 2024-09-13T11:59:09.281 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:59:09.378 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.375+0000 7fd450d97640 10 monclient: get_monmap_and_config 2024-09-13T11:59:09.378 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.375+0000 7fd450d97640 10 monclient: build_initial_monmap 2024-09-13T11:59:09.378 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.375+0000 7fd450d97640 10 monclient: monmap: 2024-09-13T11:59:09.378 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:09.378 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:09.378 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:09.377749+0000 2024-09-13T11:59:09.378 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:09.377749+0000 2024-09-13T11:59:09.378 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:09.378 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:09.379 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:09.379 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:09.379 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:09.379 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:09.379 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:09.379 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:09.386 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.376+0000 7fd450d97640 1 Processor -- start 2024-09-13T11:59:09.387 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.376+0000 7fd450d97640 1 -- start start 2024-09-13T11:59:09.387 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.376+0000 7fd450d97640 10 monclient: init 2024-09-13T11:59:09.387 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd450d97640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:09.387 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd450d97640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-13T11:59:09.387 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd450d97640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd44c0995a0 addr v1:172.21.15.122:6789/0 2024-09-13T11:59:09.387 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd450d97640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd44c09b740 0x7fd44c13c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:09.387 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd450d97640 10 monclient(hunting): picked mon.noname-b con 0x7fd44c09b740 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:09.387 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd450d97640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd44c13cc70 0x7fd44c13f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:09.387 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd450d97640 10 monclient(hunting): picked mon.noname-a con 0x7fd44c13cc70 addr v2:172.21.15.100:3300/0 2024-09-13T11:59:09.387 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd450d97640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd44c06d9a0 con 0x7fd44c0995a0 2024-09-13T11:59:09.387 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd450d97640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:09.387 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd450d97640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 -- 0x7fd44c09d350 con 0x7fd44c13cc70 2024-09-13T11:59:09.388 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd450d97640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:09.388 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd450d97640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fd44c06a330 con 0x7fd44c09b740 2024-09-13T11:59:09.388 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd450d97640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:09.388 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd450d97640 10 monclient(hunting): authenticate will time out at 1129.971197s 2024-09-13T11:59:09.388 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd449d9b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd44c09b740 0x7fd44c13c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:09.388 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd44ad9d640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd44c13cc70 0x7fd44c13f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:09.388 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd449d9b640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd44c09b740 0x7fd44c13c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:52168/0 (socket says 172.21.15.100:52168) 2024-09-13T11:59:09.388 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd449d9b640 1 -- 172.21.15.100:0/4017342119 learned_addr learned my addr 172.21.15.100:0/4017342119 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:09.388 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd449d9b640 10 monclient(hunting): get_auth_request con 0x7fd44c09b740 auth_method 0 2024-09-13T11:59:09.388 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd449d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:09.388 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd449d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:09.388 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd449d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:09.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd44ad9d640 10 monclient(hunting): get_auth_request con 0x7fd44c13cc70 auth_method 0 2024-09-13T11:59:09.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd44ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:09.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd44ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:09.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd44ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:09.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd449d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:09.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd449d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:09.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd449d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:09.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd44ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:09.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd44ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:09.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd44ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:09.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd449d9b640 10 monclient(hunting): handle_auth_done global_id 24686 payload 274 2024-09-13T11:59:09.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd449d9b640 1 -- 172.21.15.100:0/4017342119 >> v2:172.21.15.100:3300/0 conn(0x7fd44c13cc70 msgr2=0x7fd44c13f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:09.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd449d9b640 1 --2- 172.21.15.100:0/4017342119 >> v2:172.21.15.100:3300/0 conn(0x7fd44c13cc70 0x7fd44c13f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:09.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd449d9b640 1 -- 172.21.15.100:0/4017342119 >> v1:172.21.15.122:6789/0 conn(0x7fd44c0995a0 legacy=0x7fd44c099980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:09.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd449d9b640 10 monclient: _finish_hunting 0 2024-09-13T11:59:09.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd449d9b640 1 monclient: found mon.noname-b 2024-09-13T11:59:09.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd449d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:09.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd449d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:59:09.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd449d9b640 1 -- 172.21.15.100:0/4017342119 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd44c09bb20 con 0x7fd44c09b740 2024-09-13T11:59:09.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.377+0000 7fd44ad9d640 1 --2- 172.21.15.100:0/4017342119 >> v2:172.21.15.100:3300/0 conn(0x7fd44c13cc70 0x7fd44c13f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:59:09.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.378+0000 7fd449d9b640 1 --2- 172.21.15.100:0/4017342119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd44c09b740 0x7fd44c13c730 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd440009870 tx=0x7fd44002a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=351b484ccfe30708 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:09.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.378+0000 7fd44959a640 1 -- 172.21.15.100:0/4017342119 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd440037070 con 0x7fd44c09b740 2024-09-13T11:59:09.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.378+0000 7fd44959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:09.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.378+0000 7fd44959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:59:09.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.378+0000 7fd44959a640 10 monclient: dump: 2024-09-13T11:59:09.390 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:09.391 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:09.391 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:09.391 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:09.391 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:09.391 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:09.391 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:09.391 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:09.391 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:09.391 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:09.391 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.378+0000 7fd44959a640 10 monclient: _finish_auth 0 2024-09-13T11:59:09.391 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.378+0000 7fd44959a640 10 monclient: _check_auth_tickets 2024-09-13T11:59:09.391 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.378+0000 7fd44959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:09.391 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.378+0000 7fd450d97640 5 monclient: authenticate success, global_id 24686 2024-09-13T11:59:09.392 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.378+0000 7fd450d97640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:09.392 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.378+0000 7fd44959a640 1 -- 172.21.15.100:0/4017342119 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd440032440 con 0x7fd44c09b740 2024-09-13T11:59:09.392 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.378+0000 7fd44959a640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:09.392 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.378+0000 7fd450d97640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:09.392 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.378+0000 7fd450d97640 4 set_mon_vals no callback set 2024-09-13T11:59:09.392 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.378+0000 7fd44959a640 1 -- 172.21.15.100:0/4017342119 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd440032810 con 0x7fd44c09b740 2024-09-13T11:59:09.392 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.378+0000 7fd44959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:09.392 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.378+0000 7fd44959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:59:09.392 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.378+0000 7fd44959a640 10 monclient: dump: 2024-09-13T11:59:09.392 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:09.392 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:09.393 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:09.393 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:09.393 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:09.393 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:09.393 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:09.393 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:09.393 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:09.393 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:09.393 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.378+0000 7fd450d97640 10 monclient: shutdown 2024-09-13T11:59:09.393 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.378+0000 7fd450d97640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:09.393 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.378+0000 7fd450d97640 1 -- 172.21.15.100:0/4017342119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd44c09b740 msgr2=0x7fd44c13c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:09.393 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.378+0000 7fd450d97640 1 --2- 172.21.15.100:0/4017342119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd44c09b740 0x7fd44c13c730 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd440009870 tx=0x7fd44002a130 comp rx=0 tx=0).stop 2024-09-13T11:59:09.394 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.379+0000 7fd450d97640 1 -- 172.21.15.100:0/4017342119 shutdown_connections 2024-09-13T11:59:09.394 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.379+0000 7fd450d97640 1 --2- 172.21.15.100:0/4017342119 >> v2:172.21.15.100:3300/0 conn(0x7fd44c13cc70 0x7fd44c13f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:09.394 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.379+0000 7fd450d97640 1 --2- 172.21.15.100:0/4017342119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd44c09b740 0x7fd44c13c730 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:09.394 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.379+0000 7fd450d97640 1 -- 172.21.15.100:0/4017342119 >> 172.21.15.100:0/4017342119 conn(0x7fd44c0a2e70 msgr2=0x7fd44c0a3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:09.394 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.379+0000 7fd450d97640 1 -- 172.21.15.100:0/4017342119 shutdown_connections 2024-09-13T11:59:09.394 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.379+0000 7fd450d97640 1 -- 172.21.15.100:0/4017342119 wait complete. 2024-09-13T11:59:09.394 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.379+0000 7fd450d97640 10 monclient: build_initial_monmap 2024-09-13T11:59:09.394 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.379+0000 7fd450d97640 10 monclient: monmap: 2024-09-13T11:59:09.394 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:09.394 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:09.394 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:09.394 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:09.395 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:09.395 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:09.395 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:09.395 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:09.395 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:09.395 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:09.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd450d97640 1 Processor -- start 2024-09-13T11:59:09.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd450d97640 1 -- start start 2024-09-13T11:59:09.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd450d97640 10 monclient: init 2024-09-13T11:59:09.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd450d97640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:09.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd450d97640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:59:09.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd450d97640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd44c0995a0 0x7fd44c0958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:09.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd450d97640 10 monclient(hunting): picked mon.noname-c con 0x7fd44c0995a0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:09.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd450d97640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd44c09b740 0x7fd44c08e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:09.396 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd450d97640 10 monclient(hunting): picked mon.noname-b con 0x7fd44c09b740 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:09.396 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd450d97640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd44c13cc70 0x7fd44c08eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:09.396 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd450d97640 10 monclient(hunting): picked mon.noname-a con 0x7fd44c13cc70 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:09.396 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd450d97640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:09.396 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd450d97640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c06d5f0 con 0x7fd44c13cc70 2024-09-13T11:59:09.396 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd450d97640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:09.396 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd450d97640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c06d470 con 0x7fd44c09b740 2024-09-13T11:59:09.396 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd450d97640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:09.396 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd450d97640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7fd44c13f5a0 con 0x7fd44c0995a0 2024-09-13T11:59:09.396 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd450d97640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:09.396 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd450d97640 10 monclient(hunting): authenticate will time out at 1129.974603s 2024-09-13T11:59:09.396 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44a59c640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd44c0995a0 0x7fd44c0958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:09.396 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44ad9d640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd44c13cc70 0x7fd44c08eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:09.396 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44a59c640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd44c0995a0 0x7fd44c0958e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:52180/0 (socket says 172.21.15.100:52180) 2024-09-13T11:59:09.397 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44ad9d640 10 monclient(hunting): get_auth_request con 0x7fd44c13cc70 auth_method 0 2024-09-13T11:59:09.397 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44a59c640 1 -- 172.21.15.100:0/273608005 learned_addr learned my addr 172.21.15.100:0/273608005 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:09.397 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:09.397 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:09.397 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:09.397 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44a59c640 10 monclient(hunting): get_auth_request con 0x7fd44c0995a0 auth_method 0 2024-09-13T11:59:09.397 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:09.397 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44a59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:09.397 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:09.397 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd449d9b640 1 --2- 172.21.15.100:0/273608005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd44c09b740 0x7fd44c08e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:09.397 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd449d9b640 10 monclient(hunting): get_auth_request con 0x7fd44c09b740 auth_method 0 2024-09-13T11:59:09.397 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd449d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:09.397 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd449d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:09.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd449d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:09.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:09.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:09.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:09.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:09.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:09.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:09.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd449d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:09.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd449d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:09.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd449d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:09.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44ad9d640 10 monclient(hunting): handle_auth_done global_id 15603 payload 824 2024-09-13T11:59:09.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44ad9d640 1 -- 172.21.15.100:0/273608005 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd44c0995a0 msgr2=0x7fd44c0958e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:09.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44ad9d640 1 --2- 172.21.15.100:0/273608005 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd44c0995a0 0x7fd44c0958e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:09.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44ad9d640 1 -- 172.21.15.100:0/273608005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd44c09b740 msgr2=0x7fd44c08e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:09.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44ad9d640 1 --2- 172.21.15.100:0/273608005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd44c09b740 0x7fd44c08e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:09.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44ad9d640 10 monclient: _finish_hunting 0 2024-09-13T11:59:09.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44ad9d640 1 monclient: found mon.noname-a 2024-09-13T11:59:09.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:09.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:09.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.380+0000 7fd44ad9d640 1 -- 172.21.15.100:0/273608005 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd44c0a0f00 con 0x7fd44c13cc70 2024-09-13T11:59:09.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.381+0000 7fd44ad9d640 1 --2- 172.21.15.100:0/273608005 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd44c13cc70 0x7fd44c08eba0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd44c09ab10 tx=0x7fd44400ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:09.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.381+0000 7fd43affd640 1 -- 172.21.15.100:0/273608005 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd444018020 con 0x7fd44c13cc70 2024-09-13T11:59:09.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.381+0000 7fd43affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:09.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.381+0000 7fd43affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:09.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.381+0000 7fd43affd640 10 monclient: dump: 2024-09-13T11:59:09.399 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:09.400 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:09.400 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:09.400 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:09.400 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:09.400 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:09.400 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:09.400 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:09.400 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:09.400 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:09.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.381+0000 7fd43affd640 10 monclient: _finish_auth 0 2024-09-13T11:59:09.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.381+0000 7fd43affd640 10 monclient: _check_auth_tickets 2024-09-13T11:59:09.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.381+0000 7fd43affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:09.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.381+0000 7fd450d97640 5 monclient: authenticate success, global_id 15603 2024-09-13T11:59:09.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.381+0000 7fd43affd640 1 -- 172.21.15.100:0/273608005 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd444014480 con 0x7fd44c13cc70 2024-09-13T11:59:09.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.381+0000 7fd450d97640 10 monclient: _renew_subs 2024-09-13T11:59:09.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.381+0000 7fd450d97640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:09.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.381+0000 7fd450d97640 1 -- 172.21.15.100:0/273608005 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd44c090030 con 0x7fd44c13cc70 2024-09-13T11:59:09.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.381+0000 7fd43affd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:09.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.381+0000 7fd43affd640 1 -- 172.21.15.100:0/273608005 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd444014870 con 0x7fd44c13cc70 2024-09-13T11:59:09.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.381+0000 7fd43affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:09.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.381+0000 7fd43affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:09.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.381+0000 7fd43affd640 10 monclient: dump: 2024-09-13T11:59:09.401 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:09.401 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:09.401 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:09.401 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:09.401 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:09.402 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:09.402 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:09.402 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:09.402 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:09.402 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:09.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.381+0000 7fd448d99640 4 set_mon_vals no callback set 2024-09-13T11:59:09.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.381+0000 7fd450d97640 10 monclient: _renew_subs 2024-09-13T11:59:09.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.381+0000 7fd450d97640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:09.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.381+0000 7fd450d97640 1 -- 172.21.15.100:0/273608005 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd44c13ca00 con 0x7fd44c13cc70 2024-09-13T11:59:09.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.382+0000 7fd450d97640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:09.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.382+0000 7fd450d97640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:09.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.382+0000 7fd43affd640 1 -- 172.21.15.100:0/273608005 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7fd444005d10 con 0x7fd44c13cc70 2024-09-13T11:59:09.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.382+0000 7fd450d97640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:09.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.382+0000 7fd450d97640 1 -- 172.21.15.100:0/273608005 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd44c0a0c70 con 0x7fd44c13cc70 2024-09-13T11:59:09.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.382+0000 7fd43affd640 1 --2- 172.21.15.100:0/273608005 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fd42c03dfa0 0x7fd42c040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:09.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.382+0000 7fd43affd640 1 -- 172.21.15.100:0/273608005 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7fd444053470 con 0x7fd44c13cc70 2024-09-13T11:59:09.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.382+0000 7fd44a59c640 1 --2- 172.21.15.100:0/273608005 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fd42c03dfa0 0x7fd42c040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:09.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.382+0000 7fd44a59c640 10 monclient: get_auth_request con 0x7fd42c03dfa0 auth_method 0 2024-09-13T11:59:09.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.383+0000 7fd44a59c640 1 --2- 172.21.15.100:0/273608005 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fd42c03dfa0 0x7fd42c040460 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd43c007fd0 tx=0x7fd43c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:09.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.387+0000 7fd43affd640 1 -- 172.21.15.100:0/273608005 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fd44c0a0c70 con 0x7fd44c13cc70 2024-09-13T11:59:09.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.387+0000 7fd43affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:09.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.387+0000 7fd43affd640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:59:09.558 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.556+0000 7fd450d97640 1 -- 172.21.15.100:0/273608005 --> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd44c099980 con 0x7fd42c03dfa0 2024-09-13T11:59:09.559 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.557+0000 7fd43affd640 1 -- 172.21.15.100:0/273608005 <== mgr.4114 v2:172.21.15.122:6824/441789830 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34013 (secure 0 0 0) 0x7fd44c099980 con 0x7fd42c03dfa0 2024-09-13T11:59:09.560 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:59:09.560 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-13T11:59:09.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.560+0000 7fd450d97640 1 -- 172.21.15.100:0/273608005 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fd42c03dfa0 msgr2=0x7fd42c040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:09.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.560+0000 7fd450d97640 1 --2- 172.21.15.100:0/273608005 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fd42c03dfa0 0x7fd42c040460 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd43c007fd0 tx=0x7fd43c009040 comp rx=0 tx=0).stop 2024-09-13T11:59:09.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.560+0000 7fd450d97640 10 monclient: shutdown 2024-09-13T11:59:09.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.560+0000 7fd450d97640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:09.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.560+0000 7fd450d97640 1 -- 172.21.15.100:0/273608005 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd44c13cc70 msgr2=0x7fd44c08eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:09.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.560+0000 7fd450d97640 1 --2- 172.21.15.100:0/273608005 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd44c13cc70 0x7fd44c08eba0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd44c09ab10 tx=0x7fd44400ef90 comp rx=0 tx=0).stop 2024-09-13T11:59:09.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.560+0000 7fd450d97640 1 -- 172.21.15.100:0/273608005 shutdown_connections 2024-09-13T11:59:09.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.560+0000 7fd450d97640 1 --2- 172.21.15.100:0/273608005 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7fd42c03dfa0 0x7fd42c040460 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:09.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.560+0000 7fd450d97640 1 --2- 172.21.15.100:0/273608005 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd44c13cc70 0x7fd44c08eba0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:09.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.560+0000 7fd450d97640 1 --2- 172.21.15.100:0/273608005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd44c09b740 0x7fd44c08e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:09.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.560+0000 7fd450d97640 1 --2- 172.21.15.100:0/273608005 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd44c0995a0 0x7fd44c0958e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:09.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.560+0000 7fd450d97640 1 -- 172.21.15.100:0/273608005 >> 172.21.15.100:0/273608005 conn(0x7fd44c0a2e70 msgr2=0x7fd44c097fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:09.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.560+0000 7fd450d97640 1 -- 172.21.15.100:0/273608005 shutdown_connections 2024-09-13T11:59:09.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:09.560+0000 7fd450d97640 1 -- 172.21.15.100:0/273608005 wait complete. 2024-09-13T11:59:09.572 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":495,"stamp":"2024-09-13T11:59:09.287061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":211864,"kb_used_data":24728,"kb_used_omap":9,"kb_used_meta":186998,"kb_avail":566019176,"statfs":{"total":579820584960,"available":579603636224,"internally_reserved":0,"allocated":25321472,"data_stored":10410820,"data_compressed":59246,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":191486624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002661"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":599,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.972852+0000","last_change":"2024-09-13T11:53:06.266756+0000","last_active":"2024-09-13T11:58:48.972852+0000","last_peered":"2024-09-13T11:58:48.972852+0000","last_clean":"2024-09-13T11:58:48.972852+0000","last_became_active":"2024-09-13T11:53:04.566167+0000","last_became_peered":"2024-09-13T11:53:04.566167+0000","last_unstale":"2024-09-13T11:58:48.972852+0000","last_undegraded":"2024-09-13T11:58:48.972852+0000","last_fullsized":"2024-09-13T11:58:48.972852+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:41.983816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084363899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":589,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.070903+0000","last_change":"2024-09-13T11:53:06.266346+0000","last_active":"2024-09-13T11:58:48.070903+0000","last_peered":"2024-09-13T11:58:48.070903+0000","last_clean":"2024-09-13T11:58:48.070903+0000","last_became_active":"2024-09-13T11:53:04.258975+0000","last_became_peered":"2024-09-13T11:53:04.258975+0000","last_unstale":"2024-09-13T11:58:48.070903+0000","last_undegraded":"2024-09-13T11:58:48.070903+0000","last_fullsized":"2024-09-13T11:58:48.070903+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:36.773393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055464299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":599,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.972511+0000","last_change":"2024-09-13T11:53:06.266446+0000","last_active":"2024-09-13T11:58:48.972511+0000","last_peered":"2024-09-13T11:58:48.972511+0000","last_clean":"2024-09-13T11:58:48.972511+0000","last_became_active":"2024-09-13T11:53:04.262475+0000","last_became_peered":"2024-09-13T11:53:04.262475+0000","last_unstale":"2024-09-13T11:58:48.972511+0000","last_undegraded":"2024-09-13T11:58:48.972511+0000","last_fullsized":"2024-09-13T11:58:48.972511+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:24:12.639511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055213099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":589,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.068856+0000","last_change":"2024-09-13T11:53:06.266390+0000","last_active":"2024-09-13T11:58:48.068856+0000","last_peered":"2024-09-13T11:58:48.068856+0000","last_clean":"2024-09-13T11:58:48.068856+0000","last_became_active":"2024-09-13T11:53:04.262392+0000","last_became_peered":"2024-09-13T11:53:04.262392+0000","last_unstale":"2024-09-13T11:58:48.068856+0000","last_undegraded":"2024-09-13T11:58:48.068856+0000","last_fullsized":"2024-09-13T11:58:48.068856+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:45.666214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057483700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":587,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.896615+0000","last_change":"2024-09-13T11:53:06.272635+0000","last_active":"2024-09-13T11:58:48.896615+0000","last_peered":"2024-09-13T11:58:48.896615+0000","last_clean":"2024-09-13T11:58:48.896615+0000","last_became_active":"2024-09-13T11:53:04.264659+0000","last_became_peered":"2024-09-13T11:53:04.264659+0000","last_unstale":"2024-09-13T11:58:48.896615+0000","last_undegraded":"2024-09-13T11:58:48.896615+0000","last_fullsized":"2024-09-13T11:58:48.896615+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:53:49.774126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010464509999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":596,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.670015+0000","last_change":"2024-09-13T11:53:06.266298+0000","last_active":"2024-09-13T11:58:48.670015+0000","last_peered":"2024-09-13T11:58:48.670015+0000","last_clean":"2024-09-13T11:58:48.670015+0000","last_became_active":"2024-09-13T11:53:04.270146+0000","last_became_peered":"2024-09-13T11:53:04.270146+0000","last_unstale":"2024-09-13T11:58:48.670015+0000","last_undegraded":"2024-09-13T11:58:48.670015+0000","last_fullsized":"2024-09-13T11:58:48.670015+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:38.439658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061806000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":600,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.972543+0000","last_change":"2024-09-13T11:53:06.266345+0000","last_active":"2024-09-13T11:58:48.972543+0000","last_peered":"2024-09-13T11:58:48.972543+0000","last_clean":"2024-09-13T11:58:48.972543+0000","last_became_active":"2024-09-13T11:53:04.262197+0000","last_became_peered":"2024-09-13T11:53:04.262197+0000","last_unstale":"2024-09-13T11:58:48.972543+0000","last_undegraded":"2024-09-13T11:58:48.972543+0000","last_fullsized":"2024-09-13T11:58:48.972543+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:05:51.189944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044149899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":586,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.896726+0000","last_change":"2024-09-13T11:53:06.272071+0000","last_active":"2024-09-13T11:58:48.896726+0000","last_peered":"2024-09-13T11:58:48.896726+0000","last_clean":"2024-09-13T11:58:48.896726+0000","last_became_active":"2024-09-13T11:53:04.270138+0000","last_became_peered":"2024-09-13T11:53:04.270138+0000","last_unstale":"2024-09-13T11:58:48.896726+0000","last_undegraded":"2024-09-13T11:58:48.896726+0000","last_fullsized":"2024-09-13T11:58:48.896726+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:53:03.248837+0000","last_clean_scrub_stamp":"2024-09-13T11:53:03.248837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:17.090753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047405,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":691,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:58:48.973102+0000","last_change":"2024-09-13T11:52:57.706946+0000","last_active":"2024-09-13T11:58:48.973102+0000","last_peered":"2024-09-13T11:58:48.973102+0000","last_clean":"2024-09-13T11:58:48.973102+0000","last_became_active":"2024-09-13T11:52:57.705894+0000","last_became_peered":"2024-09-13T11:52:57.705894+0000","last_unstale":"2024-09-13T11:58:48.973102+0000","last_undegraded":"2024-09-13T11:58:48.973102+0000","last_fullsized":"2024-09-13T11:58:48.973102+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:52:56.225375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:52:56.225375+0000","last_clean_scrub_stamp":"2024-09-13T11:52:56.225375+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:05.299261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59246,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607630,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31128,"kb_used_data":4100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340712,"statfs":{"total":96636764160,"available":96604889088,"internally_reserved":0,"allocated":4198400,"data_stored":1558821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607629,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35348,"kb_used_data":4160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336492,"statfs":{"total":96636764160,"available":96600567808,"internally_reserved":0,"allocated":4259840,"data_stored":2087749,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607631,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35348,"kb_used_data":4160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336492,"statfs":{"total":96636764160,"available":96600567808,"internally_reserved":0,"allocated":4259840,"data_stored":2087749,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640334,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31136,"kb_used_data":4104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340704,"statfs":{"total":96636764160,"available":96604880896,"internally_reserved":0,"allocated":4202496,"data_stored":1558840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640333,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31132,"kb_used_data":4100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340708,"statfs":{"total":96636764160,"available":96604884992,"internally_reserved":0,"allocated":4198400,"data_stored":1558821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640334,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47772,"kb_used_data":4104,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94324068,"statfs":{"total":96636764160,"available":96587845632,"internally_reserved":0,"allocated":4202496,"data_stored":1558840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:59:09.576 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-13T11:53:03.248837+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=53, tm_sec=3, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=59, tm_sec=5, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:59:09.576 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-13T11:53:03.248837+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=53, tm_sec=3, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=59, tm_sec=5, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:59:09.576 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-13T11:53:03.248837+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=53, tm_sec=3, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=59, tm_sec=5, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:59:09.576 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-13T11:53:03.248837+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=53, tm_sec=3, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=59, tm_sec=5, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:59:09.576 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-13T11:53:03.248837+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=53, tm_sec=3, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=59, tm_sec=5, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:59:09.576 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-13T11:53:03.248837+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=53, tm_sec=3, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=59, tm_sec=5, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:59:09.576 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-13T11:53:03.248837+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=53, tm_sec=3, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=59, tm_sec=5, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:59:09.577 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-13T11:53:03.248837+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=53, tm_sec=3, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=59, tm_sec=5, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:59:09.577 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-13T11:52:56.225375+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=52, tm_sec=56, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=59, tm_sec=5, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:59:09.577 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-13T11:59:29.578 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:59:29.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.687+0000 7f6a950a2640 10 monclient: get_monmap_and_config 2024-09-13T11:59:29.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.687+0000 7f6a950a2640 10 monclient: build_initial_monmap 2024-09-13T11:59:29.688 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.687+0000 7f6a950a2640 10 monclient: monmap: 2024-09-13T11:59:29.688 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:29.688 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:29.688 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:29.688049+0000 2024-09-13T11:59:29.688 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:29.688049+0000 2024-09-13T11:59:29.689 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:29.689 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:29.689 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:29.689 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:29.689 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:29.689 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:29.689 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:29.689 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:29.698 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.687+0000 7f6a950a2640 1 Processor -- start 2024-09-13T11:59:29.698 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a950a2640 1 -- start start 2024-09-13T11:59:29.698 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a950a2640 10 monclient: init 2024-09-13T11:59:29.698 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a950a2640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:29.698 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a950a2640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-13T11:59:29.698 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a950a2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6a90137a60 0x7f6a900a2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:29.698 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a950a2640 10 monclient(hunting): picked mon.noname-c con 0x7f6a90137a60 addr v2:172.21.15.122:3300/0 2024-09-13T11:59:29.698 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a950a2640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a90137e40 0x7f6a900a4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:29.698 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a950a2640 10 monclient(hunting): picked mon.noname-b con 0x7f6a90137e40 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:29.698 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a950a2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6a90138270 addr v1:172.21.15.100:6789/0 2024-09-13T11:59:29.698 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a950a2640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a900a5be0 con 0x7f6a90138270 2024-09-13T11:59:29.698 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a950a2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:29.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a950a2640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 -- 0x7f6a900a4410 con 0x7f6a90137a60 2024-09-13T11:59:29.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a950a2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:29.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a950a2640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f6a900a76b0 con 0x7f6a90137e40 2024-09-13T11:59:29.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a950a2640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:29.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a950a2640 10 monclient(hunting): authenticate will time out at 1150.281619s 2024-09-13T11:59:29.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a87fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a90137e40 0x7f6a900a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:29.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a8f59e640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f6a90138270 0x7f6a900a4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:58008/0 (socket says 172.21.15.100:58008) 2024-09-13T11:59:29.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a8f59e640 1 -- 172.21.15.100:0/2166376546 learned_addr learned my addr 172.21.15.100:0/2166376546 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:59:29.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a87fff640 10 monclient(hunting): get_auth_request con 0x7f6a90137e40 auth_method 0 2024-09-13T11:59:29.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:29.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a87fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:29.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:29.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:29.700 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:29.700 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:29.700 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a8ed9d640 1 --2- 172.21.15.100:0/2166376546 >> v2:172.21.15.122:3300/0 conn(0x7f6a90137a60 0x7f6a900a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:29.700 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a8ed9d640 10 monclient(hunting): get_auth_request con 0x7f6a90137a60 auth_method 0 2024-09-13T11:59:29.700 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a8ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:29.700 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a8ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:29.700 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.688+0000 7f6a8ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:29.700 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a87fff640 10 monclient(hunting): handle_auth_done global_id 24692 payload 274 2024-09-13T11:59:29.700 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a87fff640 1 -- 172.21.15.100:0/2166376546 >> v2:172.21.15.122:3300/0 conn(0x7f6a90137a60 msgr2=0x7f6a900a2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:59:29.700 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a87fff640 1 --2- 172.21.15.100:0/2166376546 >> v2:172.21.15.122:3300/0 conn(0x7f6a90137a60 0x7f6a900a2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:29.700 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a87fff640 1 -- 172.21.15.100:0/2166376546 >> v1:172.21.15.100:6789/0 conn(0x7f6a90138270 legacy=0x7f6a900a4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:29.700 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a87fff640 10 monclient: _finish_hunting 0 2024-09-13T11:59:29.700 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a87fff640 1 monclient: found mon.noname-b 2024-09-13T11:59:29.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:29.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a87fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:59:29.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a8e59c640 1 -- 172.21.15.100:0/2166376546 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2019727476 0 0) 0x7f6a900a5be0 con 0x7f6a90138270 2024-09-13T11:59:29.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a87fff640 1 -- 172.21.15.100:0/2166376546 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a900a19e0 con 0x7f6a90137e40 2024-09-13T11:59:29.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a8e59c640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:59:29.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a8ed9d640 1 --2- 172.21.15.100:0/2166376546 >> v2:172.21.15.122:3300/0 conn(0x7f6a90137a60 0x7f6a900a2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:59:29.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a87fff640 1 --2- 172.21.15.100:0/2166376546 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a90137e40 0x7f6a900a4950 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6a900a7880 tx=0x7f6a7802a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8949195c171dc0f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:29.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a8e59c640 1 -- 172.21.15.100:0/2166376546 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a78037070 con 0x7f6a90137e40 2024-09-13T11:59:29.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a8e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:29.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a8e59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:59:29.701 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a8e59c640 10 monclient: dump: 2024-09-13T11:59:29.701 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:29.701 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:29.702 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:29.702 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:29.702 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:29.702 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:29.702 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:29.702 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:29.702 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:29.702 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:29.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a8e59c640 10 monclient: _finish_auth 0 2024-09-13T11:59:29.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a8e59c640 10 monclient: _check_auth_tickets 2024-09-13T11:59:29.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a8e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:29.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a8e59c640 1 -- 172.21.15.100:0/2166376546 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a7802adb0 con 0x7f6a90137e40 2024-09-13T11:59:29.702 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a950a2640 5 monclient: authenticate success, global_id 24692 2024-09-13T11:59:29.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a950a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:29.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a8e59c640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:29.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a8e59c640 1 -- 172.21.15.100:0/2166376546 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a78032660 con 0x7f6a90137e40 2024-09-13T11:59:29.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a8e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:29.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a8e59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:59:29.703 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.689+0000 7f6a8e59c640 10 monclient: dump: 2024-09-13T11:59:29.703 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:29.703 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:29.703 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:29.703 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:29.703 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:29.703 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:29.703 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:29.704 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:29.704 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:29.704 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:29.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.690+0000 7f6a950a2640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:29.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.690+0000 7f6a950a2640 4 set_mon_vals no callback set 2024-09-13T11:59:29.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.690+0000 7f6a950a2640 10 monclient: shutdown 2024-09-13T11:59:29.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.690+0000 7f6a950a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:29.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.690+0000 7f6a950a2640 1 -- 172.21.15.100:0/2166376546 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a90137e40 msgr2=0x7f6a900a4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:29.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.690+0000 7f6a950a2640 1 --2- 172.21.15.100:0/2166376546 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a90137e40 0x7f6a900a4950 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6a900a7880 tx=0x7f6a7802a060 comp rx=0 tx=0).stop 2024-09-13T11:59:29.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.690+0000 7f6a950a2640 1 -- 172.21.15.100:0/2166376546 shutdown_connections 2024-09-13T11:59:29.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.690+0000 7f6a950a2640 1 --2- 172.21.15.100:0/2166376546 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a90137e40 0x7f6a900a4950 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:29.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.690+0000 7f6a950a2640 1 --2- 172.21.15.100:0/2166376546 >> v2:172.21.15.122:3300/0 conn(0x7f6a90137a60 0x7f6a900a2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:29.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.690+0000 7f6a950a2640 1 -- 172.21.15.100:0/2166376546 >> 172.21.15.100:0/2166376546 conn(0x7f6a900aa580 msgr2=0x7f6a900a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:29.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.690+0000 7f6a950a2640 1 -- 172.21.15.100:0/2166376546 shutdown_connections 2024-09-13T11:59:29.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.690+0000 7f6a950a2640 1 -- 172.21.15.100:0/2166376546 wait complete. 2024-09-13T11:59:29.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.691+0000 7f6a950a2640 10 monclient: build_initial_monmap 2024-09-13T11:59:29.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.691+0000 7f6a950a2640 10 monclient: monmap: 2024-09-13T11:59:29.705 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:29.705 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:29.705 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:29.705 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:29.705 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:29.705 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:29.705 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:29.705 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:29.706 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:29.706 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:29.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.691+0000 7f6a950a2640 1 Processor -- start 2024-09-13T11:59:29.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.691+0000 7f6a950a2640 1 -- start start 2024-09-13T11:59:29.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.691+0000 7f6a950a2640 10 monclient: init 2024-09-13T11:59:29.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a950a2640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:29.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a950a2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:59:29.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a950a2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a90137a60 0x7f6a901366e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:29.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a950a2640 10 monclient(hunting): picked mon.noname-b con 0x7f6a90137a60 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:29.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a950a2640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a90137e40 0x7f6a9013c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:29.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a950a2640 10 monclient(hunting): picked mon.noname-c con 0x7f6a90137e40 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:29.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a950a2640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a90138270 0x7f6a9013cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:29.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a950a2640 10 monclient(hunting): picked mon.noname-a con 0x7f6a90138270 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:29.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a950a2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:29.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a950a2640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9006ff00 con 0x7f6a90138270 2024-09-13T11:59:29.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a950a2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:29.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a950a2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9006fd80 con 0x7f6a90137a60 2024-09-13T11:59:29.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a950a2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:29.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a950a2640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f6a900a55a0 con 0x7f6a90137e40 2024-09-13T11:59:29.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a950a2640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:29.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a950a2640 10 monclient(hunting): authenticate will time out at 1150.285306s 2024-09-13T11:59:29.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a87fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a90137e40 0x7f6a9013c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:29.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a87fff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a90137e40 0x7f6a9013c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:40740/0 (socket says 172.21.15.100:40740) 2024-09-13T11:59:29.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a87fff640 1 -- 172.21.15.100:0/3130212347 learned_addr learned my addr 172.21.15.100:0/3130212347 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:29.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a87fff640 10 monclient(hunting): get_auth_request con 0x7f6a90137e40 auth_method 0 2024-09-13T11:59:29.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:29.707 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a87fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:29.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:29.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a8f59e640 1 --2- 172.21.15.100:0/3130212347 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a90138270 0x7f6a9013cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:29.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a8f59e640 10 monclient(hunting): get_auth_request con 0x7f6a90138270 auth_method 0 2024-09-13T11:59:29.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a8f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:29.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a8f59e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:29.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a8f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:29.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:29.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:29.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:29.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a8f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:29.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a8f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:29.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a8f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:29.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a8ed9d640 1 --2- 172.21.15.100:0/3130212347 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a90137a60 0x7f6a901366e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:29.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a8ed9d640 10 monclient(hunting): get_auth_request con 0x7f6a90137a60 auth_method 0 2024-09-13T11:59:29.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a8ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:29.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a8ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:29.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a8ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:29.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a87fff640 10 monclient(hunting): handle_auth_done global_id 24695 payload 824 2024-09-13T11:59:29.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a87fff640 1 -- 172.21.15.100:0/3130212347 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a90137a60 msgr2=0x7f6a901366e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:29.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a87fff640 1 --2- 172.21.15.100:0/3130212347 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a90137a60 0x7f6a901366e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:29.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a87fff640 1 -- 172.21.15.100:0/3130212347 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a90138270 msgr2=0x7f6a9013cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:29.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a87fff640 1 --2- 172.21.15.100:0/3130212347 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a90138270 0x7f6a9013cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:29.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a87fff640 10 monclient: _finish_hunting 0 2024-09-13T11:59:29.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a87fff640 1 monclient: found mon.noname-c 2024-09-13T11:59:29.709 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:29.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a87fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.100:3301/0 2024-09-13T11:59:29.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a87fff640 1 -- 172.21.15.100:0/3130212347 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a900a8550 con 0x7f6a90137e40 2024-09-13T11:59:29.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a8f59e640 1 --2- 172.21.15.100:0/3130212347 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a90138270 0x7f6a9013cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:59:29.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.692+0000 7f6a87fff640 1 --2- 172.21.15.100:0/3130212347 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a90137e40 0x7f6a9013c670 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6a78009890 tx=0x7f6a780046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:29.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.693+0000 7f6a877fe640 1 -- 172.21.15.100:0/3130212347 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a78041020 con 0x7f6a90137e40 2024-09-13T11:59:29.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.693+0000 7f6a877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:29.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.693+0000 7f6a877fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:59:29.710 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.693+0000 7f6a877fe640 10 monclient: dump: 2024-09-13T11:59:29.710 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:29.710 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:29.710 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:29.710 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:29.710 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:29.710 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:29.711 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:29.711 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:29.711 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:29.711 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:29.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.693+0000 7f6a877fe640 10 monclient: _finish_auth 0 2024-09-13T11:59:29.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.693+0000 7f6a877fe640 10 monclient: _check_auth_tickets 2024-09-13T11:59:29.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.693+0000 7f6a877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:29.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.693+0000 7f6a877fe640 1 -- 172.21.15.100:0/3130212347 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a7803c670 con 0x7f6a90137e40 2024-09-13T11:59:29.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.693+0000 7f6a950a2640 5 monclient: authenticate success, global_id 24695 2024-09-13T11:59:29.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.693+0000 7f6a877fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:29.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.693+0000 7f6a877fe640 1 -- 172.21.15.100:0/3130212347 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a78037040 con 0x7f6a90137e40 2024-09-13T11:59:29.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.693+0000 7f6a950a2640 10 monclient: _renew_subs 2024-09-13T11:59:29.711 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.693+0000 7f6a950a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:59:29.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.693+0000 7f6a950a2640 1 -- 172.21.15.100:0/3130212347 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a90121140 con 0x7f6a90137e40 2024-09-13T11:59:29.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.693+0000 7f6a877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:29.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.693+0000 7f6a877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:59:29.712 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.693+0000 7f6a877fe640 10 monclient: dump: 2024-09-13T11:59:29.712 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:29.712 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:29.712 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:29.712 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:29.712 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:29.712 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:29.712 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:29.712 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:29.712 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:29.713 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:29.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.693+0000 7f6a8dd9b640 4 set_mon_vals no callback set 2024-09-13T11:59:29.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.693+0000 7f6a950a2640 10 monclient: _renew_subs 2024-09-13T11:59:29.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.693+0000 7f6a950a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:59:29.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.693+0000 7f6a950a2640 1 -- 172.21.15.100:0/3130212347 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a900a1140 con 0x7f6a90137e40 2024-09-13T11:59:29.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.694+0000 7f6a877fe640 1 -- 172.21.15.100:0/3130212347 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f6a78044020 con 0x7f6a90137e40 2024-09-13T11:59:29.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.694+0000 7f6a950a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:29.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.694+0000 7f6a950a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:29.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.694+0000 7f6a950a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.100:3301/0 2024-09-13T11:59:29.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.694+0000 7f6a950a2640 1 -- 172.21.15.100:0/3130212347 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a900a2980 con 0x7f6a90137e40 2024-09-13T11:59:29.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.694+0000 7f6a877fe640 1 --2- 172.21.15.100:0/3130212347 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f6a7003dfd0 0x7f6a70040490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:29.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.694+0000 7f6a877fe640 1 -- 172.21.15.100:0/3130212347 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7f6a78072cb0 con 0x7f6a90137e40 2024-09-13T11:59:29.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.694+0000 7f6a8ed9d640 1 --2- 172.21.15.100:0/3130212347 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f6a7003dfd0 0x7f6a70040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:29.713 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.694+0000 7f6a8ed9d640 10 monclient: get_auth_request con 0x7f6a7003dfd0 auth_method 0 2024-09-13T11:59:29.714 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.695+0000 7f6a8ed9d640 1 --2- 172.21.15.100:0/3130212347 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f6a7003dfd0 0x7f6a70040490 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6a80006fd0 tx=0x7f6a80008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:29.714 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.699+0000 7f6a877fe640 1 -- 172.21.15.100:0/3130212347 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f6a7804b030 con 0x7f6a90137e40 2024-09-13T11:59:29.714 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.699+0000 7f6a877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:29.714 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.699+0000 7f6a877fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:59:29.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.869+0000 7f6a950a2640 1 -- 172.21.15.100:0/3130212347 --> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6a9006a0f0 con 0x7f6a7003dfd0 2024-09-13T11:59:29.872 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.871+0000 7f6a877fe640 1 -- 172.21.15.100:0/3130212347 <== mgr.4114 v2:172.21.15.122:6824/441789830 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34030 (secure 0 0 0) 0x7f6a9006a0f0 con 0x7f6a7003dfd0 2024-09-13T11:59:29.872 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T11:59:29.873 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-13T11:59:29.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.873+0000 7f6a950a2640 1 -- 172.21.15.100:0/3130212347 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f6a7003dfd0 msgr2=0x7f6a70040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:29.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.873+0000 7f6a950a2640 1 --2- 172.21.15.100:0/3130212347 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f6a7003dfd0 0x7f6a70040490 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6a80006fd0 tx=0x7f6a80008040 comp rx=0 tx=0).stop 2024-09-13T11:59:29.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.873+0000 7f6a950a2640 10 monclient: shutdown 2024-09-13T11:59:29.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.873+0000 7f6a950a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:29.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.873+0000 7f6a950a2640 1 -- 172.21.15.100:0/3130212347 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a90137e40 msgr2=0x7f6a9013c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:29.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.873+0000 7f6a950a2640 1 --2- 172.21.15.100:0/3130212347 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a90137e40 0x7f6a9013c670 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6a78009890 tx=0x7f6a780046c0 comp rx=0 tx=0).stop 2024-09-13T11:59:29.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.873+0000 7f6a950a2640 1 -- 172.21.15.100:0/3130212347 shutdown_connections 2024-09-13T11:59:29.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.873+0000 7f6a950a2640 1 --2- 172.21.15.100:0/3130212347 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f6a7003dfd0 0x7f6a70040490 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:29.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.873+0000 7f6a950a2640 1 --2- 172.21.15.100:0/3130212347 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6a90138270 0x7f6a9013cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:29.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.873+0000 7f6a950a2640 1 --2- 172.21.15.100:0/3130212347 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6a90137e40 0x7f6a9013c670 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:29.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.873+0000 7f6a950a2640 1 --2- 172.21.15.100:0/3130212347 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a90137a60 0x7f6a901366e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:29.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.873+0000 7f6a950a2640 1 -- 172.21.15.100:0/3130212347 >> 172.21.15.100:0/3130212347 conn(0x7f6a900aa580 msgr2=0x7f6a900a4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:29.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.873+0000 7f6a950a2640 1 -- 172.21.15.100:0/3130212347 shutdown_connections 2024-09-13T11:59:29.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.873+0000 7f6a950a2640 1 -- 172.21.15.100:0/3130212347 wait complete. 2024-09-13T11:59:29.885 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-09-13T11:59:29.291381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":211836,"kb_used_data":24728,"kb_used_omap":9,"kb_used_meta":186998,"kb_avail":566019204,"statfs":{"total":579820584960,"available":579603664896,"internally_reserved":0,"allocated":25321472,"data_stored":10410820,"data_compressed":59246,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":191486624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002527"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":607,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:59:10.953999+0000","last_change":"2024-09-13T11:59:10.953999+0000","last_active":"2024-09-13T11:59:10.953999+0000","last_peered":"2024-09-13T11:59:10.953999+0000","last_clean":"2024-09-13T11:59:10.953999+0000","last_became_active":"2024-09-13T11:53:04.566167+0000","last_became_peered":"2024-09-13T11:53:04.566167+0000","last_unstale":"2024-09-13T11:59:10.953999+0000","last_undegraded":"2024-09-13T11:59:10.953999+0000","last_fullsized":"2024-09-13T11:59:10.953999+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:59:10.953885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:59:10.953885+0000","last_clean_scrub_stamp":"2024-09-13T11:59:10.953885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:55:51.770823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084363899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":597,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:59:09.485835+0000","last_change":"2024-09-13T11:59:09.485835+0000","last_active":"2024-09-13T11:59:09.485835+0000","last_peered":"2024-09-13T11:59:09.485835+0000","last_clean":"2024-09-13T11:59:09.485835+0000","last_became_active":"2024-09-13T11:53:04.258975+0000","last_became_peered":"2024-09-13T11:53:04.258975+0000","last_unstale":"2024-09-13T11:59:09.485835+0000","last_undegraded":"2024-09-13T11:59:09.485835+0000","last_fullsized":"2024-09-13T11:59:09.485835+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:59:09.485771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:59:09.485771+0000","last_clean_scrub_stamp":"2024-09-13T11:59:09.485771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:51.452704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055464299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":608,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:59:09.927018+0000","last_change":"2024-09-13T11:59:09.926955+0000","last_active":"2024-09-13T11:59:09.927018+0000","last_peered":"2024-09-13T11:59:09.927018+0000","last_clean":"2024-09-13T11:59:09.927018+0000","last_became_active":"2024-09-13T11:53:04.262475+0000","last_became_peered":"2024-09-13T11:53:04.262475+0000","last_unstale":"2024-09-13T11:59:09.927018+0000","last_undegraded":"2024-09-13T11:59:09.927018+0000","last_fullsized":"2024-09-13T11:59:09.927018+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:59:09.926854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:59:09.926854+0000","last_clean_scrub_stamp":"2024-09-13T11:59:09.926854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:25.033963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055213099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":598,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:59:08.470365+0000","last_change":"2024-09-13T11:59:08.470316+0000","last_active":"2024-09-13T11:59:08.470365+0000","last_peered":"2024-09-13T11:59:08.470365+0000","last_clean":"2024-09-13T11:59:08.470365+0000","last_became_active":"2024-09-13T11:53:04.262392+0000","last_became_peered":"2024-09-13T11:53:04.262392+0000","last_unstale":"2024-09-13T11:59:08.470365+0000","last_undegraded":"2024-09-13T11:59:08.470365+0000","last_fullsized":"2024-09-13T11:59:08.470365+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:59:08.470253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:59:08.470253+0000","last_clean_scrub_stamp":"2024-09-13T11:59:08.470253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:42:06.924449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057483700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"313'3","reported_seq":597,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:59:10.908215+0000","last_change":"2024-09-13T11:59:10.908146+0000","last_active":"2024-09-13T11:59:10.908215+0000","last_peered":"2024-09-13T11:59:10.908215+0000","last_clean":"2024-09-13T11:59:10.908215+0000","last_became_active":"2024-09-13T11:53:04.264659+0000","last_became_peered":"2024-09-13T11:53:04.264659+0000","last_unstale":"2024-09-13T11:59:10.908215+0000","last_undegraded":"2024-09-13T11:59:10.908215+0000","last_fullsized":"2024-09-13T11:59:10.908215+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"313'3","last_scrub_stamp":"2024-09-13T11:59:10.908070+0000","last_deep_scrub":"313'3","last_deep_scrub_stamp":"2024-09-13T11:59:10.908070+0000","last_clean_scrub_stamp":"2024-09-13T11:59:10.908070+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:08:44.684444+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010464509999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":605,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:59:08.935094+0000","last_change":"2024-09-13T11:59:08.935011+0000","last_active":"2024-09-13T11:59:08.935094+0000","last_peered":"2024-09-13T11:59:08.935094+0000","last_clean":"2024-09-13T11:59:08.935094+0000","last_became_active":"2024-09-13T11:53:04.270146+0000","last_became_peered":"2024-09-13T11:53:04.270146+0000","last_unstale":"2024-09-13T11:59:08.935094+0000","last_undegraded":"2024-09-13T11:59:08.935094+0000","last_fullsized":"2024-09-13T11:59:08.935094+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:59:08.934929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:59:08.934929+0000","last_clean_scrub_stamp":"2024-09-13T11:59:08.934929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:59.874648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061806000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":608,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:59:11.904707+0000","last_change":"2024-09-13T11:59:11.904707+0000","last_active":"2024-09-13T11:59:11.904707+0000","last_peered":"2024-09-13T11:59:11.904707+0000","last_clean":"2024-09-13T11:59:11.904707+0000","last_became_active":"2024-09-13T11:53:04.262197+0000","last_became_peered":"2024-09-13T11:53:04.262197+0000","last_unstale":"2024-09-13T11:59:11.904707+0000","last_undegraded":"2024-09-13T11:59:11.904707+0000","last_fullsized":"2024-09-13T11:59:11.904707+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:59:11.904630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:59:11.904630+0000","last_clean_scrub_stamp":"2024-09-13T11:59:11.904630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:05:15.208041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044149899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"313'2","reported_seq":596,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:59:09.925826+0000","last_change":"2024-09-13T11:59:09.925769+0000","last_active":"2024-09-13T11:59:09.925826+0000","last_peered":"2024-09-13T11:59:09.925826+0000","last_clean":"2024-09-13T11:59:09.925826+0000","last_became_active":"2024-09-13T11:53:04.270138+0000","last_became_peered":"2024-09-13T11:53:04.270138+0000","last_unstale":"2024-09-13T11:59:09.925826+0000","last_undegraded":"2024-09-13T11:59:09.925826+0000","last_fullsized":"2024-09-13T11:59:09.925826+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"313'2","last_scrub_stamp":"2024-09-13T11:59:09.925699+0000","last_deep_scrub":"313'2","last_deep_scrub_stamp":"2024-09-13T11:59:09.925699+0000","last_clean_scrub_stamp":"2024-09-13T11:59:09.925699+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:13.669155+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00047405,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"313'74","reported_seq":701,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T11:59:08.939806+0000","last_change":"2024-09-13T11:59:08.939806+0000","last_active":"2024-09-13T11:59:08.939806+0000","last_peered":"2024-09-13T11:59:08.939806+0000","last_clean":"2024-09-13T11:59:08.939806+0000","last_became_active":"2024-09-13T11:52:57.705894+0000","last_became_peered":"2024-09-13T11:52:57.705894+0000","last_unstale":"2024-09-13T11:59:08.939806+0000","last_undegraded":"2024-09-13T11:59:08.939806+0000","last_fullsized":"2024-09-13T11:59:08.939806+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"313'74","last_scrub_stamp":"2024-09-13T11:59:08.939724+0000","last_deep_scrub":"313'74","last_deep_scrub_stamp":"2024-09-13T11:59:08.939724+0000","last_clean_scrub_stamp":"2024-09-13T11:59:08.939724+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:13:50.751727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59246,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607634,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31128,"kb_used_data":4100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340712,"statfs":{"total":96636764160,"available":96604889088,"internally_reserved":0,"allocated":4198400,"data_stored":1558821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607633,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35344,"kb_used_data":4160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336496,"statfs":{"total":96636764160,"available":96600571904,"internally_reserved":0,"allocated":4259840,"data_stored":2087749,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607635,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35344,"kb_used_data":4160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336496,"statfs":{"total":96636764160,"available":96600571904,"internally_reserved":0,"allocated":4259840,"data_stored":2087749,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640338,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31128,"kb_used_data":4104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340712,"statfs":{"total":96636764160,"available":96604889088,"internally_reserved":0,"allocated":4202496,"data_stored":1558840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640337,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31124,"kb_used_data":4100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340716,"statfs":{"total":96636764160,"available":96604893184,"internally_reserved":0,"allocated":4198400,"data_stored":1558821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640338,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47768,"kb_used_data":4104,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94324072,"statfs":{"total":96636764160,"available":96587849728,"internally_reserved":0,"allocated":4202496,"data_stored":1558840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29623,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:59:29.886 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-13T11:59:29.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.981+0000 7f4ec2318640 10 monclient: get_monmap_and_config 2024-09-13T11:59:29.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.981+0000 7f4ec2318640 10 monclient: build_initial_monmap 2024-09-13T11:59:29.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.981+0000 7f4ec2318640 10 monclient: monmap: 2024-09-13T11:59:29.983 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:29.983 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:29.983 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:59:29.982534+0000 2024-09-13T11:59:29.983 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:59:29.982534+0000 2024-09-13T11:59:29.983 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:29.983 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:29.983 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a-legacy 2024-09-13T11:59:29.983 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.122:6789/0 mon.noname-c-legacy 2024-09-13T11:59:29.983 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-b 2024-09-13T11:59:29.983 INFO:teuthology.orchestra.run.smithi100.stderr:3: v2:172.21.15.100:3300/0 mon.noname-a 2024-09-13T11:59:29.983 INFO:teuthology.orchestra.run.smithi100.stderr:4: v2:172.21.15.122:3300/0 mon.noname-c 2024-09-13T11:59:29.984 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:29.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.982+0000 7f4ec2318640 1 Processor -- start 2024-09-13T11:59:29.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.982+0000 7f4ec2318640 1 -- start start 2024-09-13T11:59:29.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.982+0000 7f4ec2318640 10 monclient: init 2024-09-13T11:59:29.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.982+0000 7f4ec2318640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:29.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.982+0000 7f4ec2318640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-13T11:59:29.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec2318640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ebc1357c0 addr v1:172.21.15.100:6789/0 2024-09-13T11:59:29.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec2318640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4ebc0a1f40 addr v1:172.21.15.122:6789/0 2024-09-13T11:59:29.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec2318640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4ebc13bdd0 0x7f4ebc13e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:29.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec2318640 10 monclient(hunting): picked mon.noname-b con 0x7f4ebc13bdd0 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:29.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec2318640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ebc1362b0 con 0x7f4ebc1357c0 2024-09-13T11:59:29.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec2318640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ebc074560 con 0x7f4ebc0a1f40 2024-09-13T11:59:29.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec2318640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:29.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec2318640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f4ebc0a1310 con 0x7f4ebc13bdd0 2024-09-13T11:59:29.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec2318640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:29.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec2318640 10 monclient(hunting): authenticate will time out at 1150.576127s 2024-09-13T11:59:29.992 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ebbfff640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f4ebc1357c0 0x7f4ebc135ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:58014/0 (socket says 172.21.15.100:58014) 2024-09-13T11:59:29.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ebbfff640 1 -- 172.21.15.100:0/3241286962 learned_addr learned my addr 172.21.15.100:0/3241286962 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-13T11:59:29.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec08b5640 1 --2- 172.21.15.100:0/3241286962 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4ebc13bdd0 0x7f4ebc13e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:29.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec08b5640 10 monclient(hunting): get_auth_request con 0x7f4ebc13bdd0 auth_method 0 2024-09-13T11:59:29.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec08b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:29.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec08b5640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:29.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec08b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:29.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec08b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:29.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec08b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:29.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec08b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:29.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ebaffd640 1 -- 172.21.15.100:0/3241286962 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 75106872 0 0) 0x7f4ebc1362b0 con 0x7f4ebc1357c0 2024-09-13T11:59:29.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ebaffd640 10 monclient(hunting): _negotiate 2024-09-13T11:59:29.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ebaffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:29.993 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ebaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:29.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ebaffd640 10 monclient(hunting): my global_id is 15612 2024-09-13T11:59:29.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ebaffd640 1 -- 172.21.15.100:0/3241286962 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ea4003520 con 0x7f4ebc1357c0 2024-09-13T11:59:29.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec08b5640 10 monclient(hunting): handle_auth_done global_id 24698 payload 274 2024-09-13T11:59:29.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec08b5640 1 -- 172.21.15.100:0/3241286962 >> v1:172.21.15.122:6789/0 conn(0x7f4ebc0a1f40 legacy=0x7f4ebc13a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:29.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec08b5640 1 -- 172.21.15.100:0/3241286962 >> v1:172.21.15.100:6789/0 conn(0x7f4ebc1357c0 legacy=0x7f4ebc135ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:29.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec08b5640 10 monclient: _finish_hunting 0 2024-09-13T11:59:29.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec08b5640 1 monclient: found mon.noname-b 2024-09-13T11:59:29.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec08b5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:29.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec08b5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.100:3301/0 2024-09-13T11:59:29.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.983+0000 7f4ec08b5640 1 -- 172.21.15.100:0/3241286962 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ebc09fa40 con 0x7f4ebc13bdd0 2024-09-13T11:59:29.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ec08b5640 1 --2- 172.21.15.100:0/3241286962 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4ebc13bdd0 0x7f4ebc13e1c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4009dc0 tx=0x7f4eb402a5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7bcdc2ca145239c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:29.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ebaffd640 1 -- 172.21.15.100:0/3241286962 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb4007740 con 0x7f4ebc13bdd0 2024-09-13T11:59:29.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ebaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:29.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ebaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:59:29.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ebaffd640 10 monclient: dump: 2024-09-13T11:59:29.995 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:29.995 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:29.995 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:29.995 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:29.995 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:29.995 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:29.995 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:29.995 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:29.995 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:29.995 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:29.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ebaffd640 10 monclient: _finish_auth 0 2024-09-13T11:59:29.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ebaffd640 10 monclient: _check_auth_tickets 2024-09-13T11:59:29.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ebaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:29.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ec2318640 5 monclient: authenticate success, global_id 24698 2024-09-13T11:59:29.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ebaffd640 1 -- 172.21.15.100:0/3241286962 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4eb40078e0 con 0x7f4ebc13bdd0 2024-09-13T11:59:29.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ec2318640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:59:29.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ebaffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:29.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ec2318640 10 monclient: get_monmap_and_config success 2024-09-13T11:59:29.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ec2318640 4 set_mon_vals no callback set 2024-09-13T11:59:29.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ebaffd640 1 -- 172.21.15.100:0/3241286962 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb4007cb0 con 0x7f4ebc13bdd0 2024-09-13T11:59:29.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ebaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:29.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ebaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:59:29.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ebaffd640 10 monclient: dump: 2024-09-13T11:59:29.996 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:29.997 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:29.997 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:29.997 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:29.997 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:29.997 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:29.997 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:29.997 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:29.997 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:29.997 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:29.997 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ec2318640 10 monclient: shutdown 2024-09-13T11:59:29.997 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ec2318640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:29.997 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ec2318640 1 -- 172.21.15.100:0/3241286962 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4ebc13bdd0 msgr2=0x7f4ebc13e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:29.997 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ec2318640 1 --2- 172.21.15.100:0/3241286962 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4ebc13bdd0 0x7f4ebc13e1c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4009dc0 tx=0x7f4eb402a5b0 comp rx=0 tx=0).stop 2024-09-13T11:59:29.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ec2318640 1 -- 172.21.15.100:0/3241286962 shutdown_connections 2024-09-13T11:59:29.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ec2318640 1 --2- 172.21.15.100:0/3241286962 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4ebc13bdd0 0x7f4ebc13e1c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:29.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ec2318640 1 -- 172.21.15.100:0/3241286962 >> 172.21.15.100:0/3241286962 conn(0x7f4ebc0a82e0 msgr2=0x7f4ebc0a47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:29.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.984+0000 7f4ec2318640 1 -- 172.21.15.100:0/3241286962 shutdown_connections 2024-09-13T11:59:29.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.985+0000 7f4ec2318640 1 -- 172.21.15.100:0/3241286962 wait complete. 2024-09-13T11:59:29.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.985+0000 7f4ec2318640 10 monclient: build_initial_monmap 2024-09-13T11:59:29.998 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.985+0000 7f4ec2318640 10 monclient: monmap: 2024-09-13T11:59:29.998 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-13T11:59:29.998 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:59:29.998 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-13T11:59:29.999 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-13T11:59:29.999 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-13T11:59:29.999 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:29.999 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.noname-a 2024-09-13T11:59:29.999 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.noname-b 2024-09-13T11:59:29.999 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.noname-c 2024-09-13T11:59:29.999 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:29.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.985+0000 7f4ec2318640 1 Processor -- start 2024-09-13T11:59:29.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec2318640 1 -- start start 2024-09-13T11:59:29.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec2318640 10 monclient: init 2024-09-13T11:59:29.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec2318640 10 monclient: _reopen_session rank -1 2024-09-13T11:59:29.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec2318640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:59:29.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec2318640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4ebc0a1f40 0x7f4ebc133e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:29.999 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec2318640 10 monclient(hunting): picked mon.noname-c con 0x7f4ebc0a1f40 addr [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-13T11:59:30.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec2318640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ebc1357c0 0x7f4ebc134370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:30.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec2318640 10 monclient(hunting): picked mon.noname-b con 0x7f4ebc1357c0 addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-09-13T11:59:30.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec2318640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4ebc13bdd0 0x7f4ebc137a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:30.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec2318640 10 monclient(hunting): picked mon.noname-a con 0x7f4ebc13bdd0 addr [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] 2024-09-13T11:59:30.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec2318640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:30.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec2318640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 -- 0x7f4ebc0a5130 con 0x7f4ebc13bdd0 2024-09-13T11:59:30.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec2318640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:30.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec2318640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f4ebc09fcf0 con 0x7f4ebc1357c0 2024-09-13T11:59:30.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec2318640 10 monclient(hunting): start opening mon connection 2024-09-13T11:59:30.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec2318640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 -- 0x7f4ebc0a52b0 con 0x7f4ebc0a1f40 2024-09-13T11:59:30.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec2318640 10 monclient(hunting): _renew_subs 2024-09-13T11:59:30.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec2318640 10 monclient(hunting): authenticate will time out at 1150.579453s 2024-09-13T11:59:30.000 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec08b5640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4ebc13bdd0 0x7f4ebc137a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:30.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ebbfff640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4ebc0a1f40 0x7f4ebc133e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:30.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec08b5640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4ebc13bdd0 0x7f4ebc137a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:37560/0 (socket says 172.21.15.100:37560) 2024-09-13T11:59:30.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec08b5640 1 -- 172.21.15.100:0/4207635946 learned_addr learned my addr 172.21.15.100:0/4207635946 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-09-13T11:59:30.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec08b5640 10 monclient(hunting): get_auth_request con 0x7f4ebc13bdd0 auth_method 0 2024-09-13T11:59:30.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec08b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:30.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec08b5640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:30.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec08b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:30.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ebbfff640 1 --2- 172.21.15.100:0/4207635946 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4ebc0a1f40 0x7f4ebc133e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:40764/0 (socket says 172.21.15.100:40764) 2024-09-13T11:59:30.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ebbfff640 10 monclient(hunting): get_auth_request con 0x7f4ebc0a1f40 auth_method 0 2024-09-13T11:59:30.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ebbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:59:30.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ebbfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:59:30.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ebbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:59:30.001 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec08b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:30.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec08b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:30.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec08b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:30.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ebbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:59:30.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ebbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:59:30.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ebbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:59:30.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ebb7fe640 1 --2- 172.21.15.100:0/4207635946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ebc1357c0 0x7f4ebc134370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:30.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec08b5640 10 monclient(hunting): handle_auth_done global_id 15615 payload 824 2024-09-13T11:59:30.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec08b5640 1 -- 172.21.15.100:0/4207635946 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4ebc0a1f40 msgr2=0x7f4ebc133e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:30.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec08b5640 1 --2- 172.21.15.100:0/4207635946 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4ebc0a1f40 0x7f4ebc133e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:30.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec08b5640 1 -- 172.21.15.100:0/4207635946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ebc1357c0 msgr2=0x7f4ebc134370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:30.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec08b5640 1 --2- 172.21.15.100:0/4207635946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ebc1357c0 0x7f4ebc134370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:30.002 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec08b5640 10 monclient: _finish_hunting 0 2024-09-13T11:59:30.003 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec08b5640 1 monclient: found mon.noname-a 2024-09-13T11:59:30.003 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec08b5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:59:30.003 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec08b5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.100:3300/0 2024-09-13T11:59:30.003 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec08b5640 1 -- 172.21.15.100:0/4207635946 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ebc13b460 con 0x7f4ebc13bdd0 2024-09-13T11:59:30.003 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ebb7fe640 10 monclient: get_auth_request con 0x7f4ebc1357c0 auth_method 0 2024-09-13T11:59:30.003 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ebb7fe640 1 --2- 172.21.15.100:0/4207635946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ebc1357c0 0x7f4ebc134370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:59:30.003 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.986+0000 7f4ec08b5640 1 --2- 172.21.15.100:0/4207635946 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4ebc13bdd0 0x7f4ebc137a10 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4009ee0 tx=0x7f4eb4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:30.003 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.987+0000 7f4eb8ff9640 1 -- 172.21.15.100:0/4207635946 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb4040070 con 0x7f4ebc13bdd0 2024-09-13T11:59:30.003 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.987+0000 7f4eb8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:30.003 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.987+0000 7f4eb8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:59:30.003 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.987+0000 7f4eb8ff9640 10 monclient: dump: 2024-09-13T11:59:30.003 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:30.003 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:30.004 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:30.004 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:30.004 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:30.004 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:30.004 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:30.004 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:30.004 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:30.004 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:30.004 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.987+0000 7f4eb8ff9640 10 monclient: _finish_auth 0 2024-09-13T11:59:30.004 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.987+0000 7f4eb8ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:59:30.004 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.987+0000 7f4eb8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:59:30.004 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.987+0000 7f4eb8ff9640 1 -- 172.21.15.100:0/4207635946 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4eb4031bc0 con 0x7f4ebc13bdd0 2024-09-13T11:59:30.004 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.987+0000 7f4ec2318640 5 monclient: authenticate success, global_id 15615 2024-09-13T11:59:30.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.987+0000 7f4eb8ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:59:30.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.987+0000 7f4eb8ff9640 1 -- 172.21.15.100:0/4207635946 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb4009450 con 0x7f4ebc13bdd0 2024-09-13T11:59:30.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.987+0000 7f4ec2318640 10 monclient: _renew_subs 2024-09-13T11:59:30.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.987+0000 7f4ec2318640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:30.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.987+0000 7f4ec2318640 1 -- 172.21.15.100:0/4207635946 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ebc13b740 con 0x7f4ebc13bdd0 2024-09-13T11:59:30.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.987+0000 7f4eb8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:59:30.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.987+0000 7f4eb8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:59:30.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.987+0000 7f4eb8ff9640 10 monclient: dump: 2024-09-13T11:59:30.005 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-13T11:59:30.005 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 0b1f4b4d-fad4-4dd3-86ac-a286d80846c0 2024-09-13T11:59:30.005 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:30.005 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-13T11:52:13.286120+0000 2024-09-13T11:59:30.005 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-13T11:59:30.006 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-13T11:59:30.006 INFO:teuthology.orchestra.run.smithi100.stderr:0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-09-13T11:59:30.006 INFO:teuthology.orchestra.run.smithi100.stderr:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-09-13T11:59:30.006 INFO:teuthology.orchestra.run.smithi100.stderr:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-09-13T11:59:30.006 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:30.006 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.987+0000 7f4eb9ffb640 4 set_mon_vals no callback set 2024-09-13T11:59:30.006 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.987+0000 7f4ec2318640 10 monclient: _renew_subs 2024-09-13T11:59:30.006 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.987+0000 7f4ec2318640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:30.006 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.987+0000 7f4ec2318640 1 -- 172.21.15.100:0/4207635946 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ebc076b50 con 0x7f4ebc13bdd0 2024-09-13T11:59:30.006 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.988+0000 7f4eb8ff9640 1 -- 172.21.15.100:0/4207635946 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) ==== 50419+0+0 (secure 0 0 0) 0x7f4eb4039040 con 0x7f4ebc13bdd0 2024-09-13T11:59:30.006 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.988+0000 7f4ec2318640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:30.006 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.988+0000 7f4ec2318640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:30.006 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.988+0000 7f4ec2318640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:30.006 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.988+0000 7f4ec2318640 1 -- 172.21.15.100:0/4207635946 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ebc09ff70 con 0x7f4ebc13bdd0 2024-09-13T11:59:30.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.988+0000 7f4eb8ff9640 1 --2- 172.21.15.100:0/4207635946 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f4ea803dfd0 0x7f4ea8040490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:59:30.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.988+0000 7f4eb8ff9640 1 -- 172.21.15.100:0/4207635946 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(313..313 src has 1..313) ==== 4346+0+0 (secure 0 0 0) 0x7f4eb4073de0 con 0x7f4ebc13bdd0 2024-09-13T11:59:30.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.988+0000 7f4ebbfff640 1 --2- 172.21.15.100:0/4207635946 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f4ea803dfd0 0x7f4ea8040490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:59:30.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.988+0000 7f4ebbfff640 10 monclient: get_auth_request con 0x7f4ea803dfd0 auth_method 0 2024-09-13T11:59:30.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.989+0000 7f4ebbfff640 1 --2- 172.21.15.100:0/4207635946 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f4ea803dfd0 0x7f4ea8040490 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4eac007f50 tx=0x7f4eac007ee0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:59:30.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.993+0000 7f4eb8ff9640 1 -- 172.21.15.100:0/4207635946 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f4ebc09ff70 con 0x7f4ebc13bdd0 2024-09-13T11:59:30.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.993+0000 7f4eb8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:59:30.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:29.993+0000 7f4eb8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:59:30.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.168+0000 7f4ec2318640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-13T11:59:30.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.168+0000 7f4ec2318640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-13T11:59:30.169 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.168+0000 7f4ec2318640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.100:3300/0 2024-09-13T11:59:30.170 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.168+0000 7f4ec2318640 1 -- 172.21.15.100:0/4207635946 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f4ebc135ba0 con 0x7f4ebc13bdd0 2024-09-13T11:59:30.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.173+0000 7f4eb8ff9640 1 -- 172.21.15.100:0/4207635946 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v7)=0 v7) ==== 123+0+0 (secure 0 0 0) 0x7f4eb403cbd0 con 0x7f4ebc13bdd0 2024-09-13T11:59:30.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.173+0000 7f4eb8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-13T11:59:30.174 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.173+0000 7f4eb8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:59:30.175 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.174+0000 7f4eb8ff9640 1 -- 172.21.15.100:0/4207635946 <== mon.0 v2:172.21.15.100:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f4eb403a750 con 0x7f4ebc13bdd0 2024-09-13T11:59:30.175 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.174+0000 7f4eb8ff9640 10 monclient: handle_config config(1 keys) 2024-09-13T11:59:30.175 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.174+0000 7f4eba7fc640 4 set_mon_vals no callback set 2024-09-13T11:59:30.175 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.174+0000 7f4eba7fc640 10 set_mon_vals mon_health_to_clog = false 2024-09-13T11:59:30.177 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.176+0000 7f4ec2318640 1 -- 172.21.15.100:0/4207635946 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f4ea803dfd0 msgr2=0x7f4ea8040490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:30.177 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.176+0000 7f4ec2318640 1 --2- 172.21.15.100:0/4207635946 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f4ea803dfd0 0x7f4ea8040490 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4eac007f50 tx=0x7f4eac007ee0 comp rx=0 tx=0).stop 2024-09-13T11:59:30.177 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.176+0000 7f4ec2318640 10 monclient: shutdown 2024-09-13T11:59:30.177 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.176+0000 7f4ec2318640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:59:30.178 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.176+0000 7f4ec2318640 1 -- 172.21.15.100:0/4207635946 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4ebc13bdd0 msgr2=0x7f4ebc137a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:59:30.178 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.176+0000 7f4ec2318640 1 --2- 172.21.15.100:0/4207635946 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4ebc13bdd0 0x7f4ebc137a10 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4009ee0 tx=0x7f4eb4004a40 comp rx=0 tx=0).stop 2024-09-13T11:59:30.178 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.176+0000 7f4ec2318640 1 -- 172.21.15.100:0/4207635946 shutdown_connections 2024-09-13T11:59:30.178 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.176+0000 7f4ec2318640 1 --2- 172.21.15.100:0/4207635946 >> [v2:172.21.15.122:6824/441789830,v1:172.21.15.122:6825/441789830] conn(0x7f4ea803dfd0 0x7f4ea8040490 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:30.178 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.176+0000 7f4ec2318640 1 --2- 172.21.15.100:0/4207635946 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4ebc13bdd0 0x7f4ebc137a10 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:30.178 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.176+0000 7f4ec2318640 1 --2- 172.21.15.100:0/4207635946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ebc1357c0 0x7f4ebc134370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:30.178 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.176+0000 7f4ec2318640 1 --2- 172.21.15.100:0/4207635946 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4ebc0a1f40 0x7f4ebc133e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:59:30.178 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.176+0000 7f4ec2318640 1 -- 172.21.15.100:0/4207635946 >> 172.21.15.100:0/4207635946 conn(0x7f4ebc0a82e0 msgr2=0x7f4ebc13c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:59:30.178 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.176+0000 7f4ec2318640 1 -- 172.21.15.100:0/4207635946 shutdown_connections 2024-09-13T11:59:30.178 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T11:59:30.176+0000 7f4ec2318640 1 -- 172.21.15.100:0/4207635946 wait complete. 2024-09-13T11:59:30.188 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-13T11:59:30.188 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-13T11:59:30.188 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-13T11:59:30.188 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:59:30.223 INFO:tasks.ceph.osd.0:Stopped 2024-09-13T11:59:30.223 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-13T11:59:30.223 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:59:30.293 INFO:tasks.ceph.osd.1:Stopped 2024-09-13T11:59:30.294 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-13T11:59:30.294 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:59:30.360 INFO:tasks.ceph.osd.2:Stopped 2024-09-13T11:59:30.361 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-13T11:59:30.361 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:59:30.389 INFO:tasks.ceph.osd.3:Stopped 2024-09-13T11:59:30.389 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-13T11:59:30.389 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:59:30.465 INFO:tasks.ceph.osd.4:Stopped 2024-09-13T11:59:30.465 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-13T11:59:30.465 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:59:30.538 INFO:tasks.ceph.osd.5:Stopped 2024-09-13T11:59:30.538 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-13T11:59:30.539 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-13T11:59:30.539 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:59:30.608 INFO:tasks.ceph.mgr.x:Stopped 2024-09-13T11:59:30.608 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-13T11:59:30.609 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-13T11:59:30.609 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:59:30.620 INFO:tasks.ceph.mon.a:Stopped 2024-09-13T11:59:30.620 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-13T11:59:30.621 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:59:30.672 INFO:tasks.ceph.mon.c:Stopped 2024-09-13T11:59:30.673 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-13T11:59:30.673 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:59:30.684 INFO:tasks.ceph.mon.b:Stopped 2024-09-13T11:59:30.685 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-13T11:59:30.685 DEBUG:teuthology.orchestra.run.smithi100:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | head -n 1 2024-09-13T11:59:30.742 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-13T11:59:30.743 DEBUG:teuthology.orchestra.run.smithi100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-13T11:59:30.950 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-13T11:59:30.950 DEBUG:teuthology.orchestra.run.smithi100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-13T11:59:31.023 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-13T11:59:31.024 DEBUG:teuthology.orchestra.run.smithi100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-13T11:59:31.072 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T11:59:31.072 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-13T11:59:31.199 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T11:59:31.199 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-13T11:59:31.296 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T11:59:31.297 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-13T11:59:31.344 INFO:tasks.ceph:Archiving mon data... 2024-09-13T11:59:31.345 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903450/data/mon.a.tgz 2024-09-13T11:59:31.345 DEBUG:teuthology.orchestra.run.smithi100:> mktemp 2024-09-13T11:59:31.361 INFO:teuthology.orchestra.run.smithi100.stdout:/tmp/tmp.TUUQCG14qz 2024-09-13T11:59:31.361 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.TUUQCG14qz 2024-09-13T11:59:31.514 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0666 /tmp/tmp.TUUQCG14qz 2024-09-13T11:59:31.597 DEBUG:teuthology.orchestra.remote:smithi100:/tmp/tmp.TUUQCG14qz is 366KB 2024-09-13T11:59:31.694 DEBUG:teuthology.orchestra.run.smithi100:> rm -fr /tmp/tmp.TUUQCG14qz 2024-09-13T11:59:31.709 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903450/data/mon.c.tgz 2024-09-13T11:59:31.709 DEBUG:teuthology.orchestra.run.smithi100:> mktemp 2024-09-13T11:59:31.764 INFO:teuthology.orchestra.run.smithi100.stdout:/tmp/tmp.1B6mKuOH4b 2024-09-13T11:59:31.764 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.1B6mKuOH4b 2024-09-13T11:59:31.901 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0666 /tmp/tmp.1B6mKuOH4b 2024-09-13T11:59:31.984 DEBUG:teuthology.orchestra.remote:smithi100:/tmp/tmp.1B6mKuOH4b is 366KB 2024-09-13T11:59:32.079 DEBUG:teuthology.orchestra.run.smithi100:> rm -fr /tmp/tmp.1B6mKuOH4b 2024-09-13T11:59:32.094 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903450/data/mon.b.tgz 2024-09-13T11:59:32.095 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2024-09-13T11:59:32.113 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.xmKN4jiax5 2024-09-13T11:59:32.113 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.xmKN4jiax5 2024-09-13T11:59:32.267 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.xmKN4jiax5 2024-09-13T11:59:32.352 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.xmKN4jiax5 is 364KB 2024-09-13T11:59:32.450 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.xmKN4jiax5 2024-09-13T11:59:32.465 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-13T11:59:32.465 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T11:59:32.468 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T11:59:32.546 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-13T11:59:32.585 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-13T11:59:32.585 INFO:tasks.ceph:Archiving crash dumps... 2024-09-13T11:59:32.587 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/lib/ceph/crash to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903450/remote/smithi100/crash 2024-09-13T11:59:32.587 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T11:59:32.612 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/crash to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903450/remote/smithi122/crash 2024-09-13T11:59:32.613 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T11:59:32.650 INFO:tasks.ceph:Compressing logs... 2024-09-13T11:59:32.650 DEBUG:teuthology.orchestra.run.smithi100:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T11:59:32.656 DEBUG:teuthology.orchestra.run.smithi122:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T11:59:32.677 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-13T11:59:32.677 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37766.log 2024-09-13T11:59:32.761 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37421.log 2024-09-13T11:59:32.762 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.37766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37808.log 2024-09-13T11:59:32.762 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38146.log 2024-09-13T11:59:32.762 INFO:teuthology.orchestra.run.smithi100.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.37766.log.gz 2024-09-13T11:59:32.762 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35761.log 2024-09-13T11:59:32.762 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37808.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37421.log.gz 2024-09-13T11:59:32.762 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37465.log 2024-09-13T11:59:32.762 INFO:teuthology.orchestra.run.smithi100.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.37808.log.gz 2024-09-13T11:59:32.762 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38246.log 2024-09-13T11:59:32.763 INFO:teuthology.orchestra.run.smithi100.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.38146.log.gz 2024-09-13T11:59:32.763 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34677.log 2024-09-13T11:59:32.763 INFO:teuthology.orchestra.run.smithi100.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35761.log.gz 2024-09-13T11:59:32.763 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38246.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37465.log.gz 2024-09-13T11:59:32.763 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35019.log 2024-09-13T11:59:32.763 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.34677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38372.log 2024-09-13T11:59:32.763 INFO:teuthology.orchestra.run.smithi100.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38246.log.gz 2024-09-13T11:59:32.763 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35604.log 2024-09-13T11:59:32.763 INFO:teuthology.orchestra.run.smithi100.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34677.log.gz 2024-09-13T11:59:32.763 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38372.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35019.log.gz 2024-09-13T11:59:32.764 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37377.log 2024-09-13T11:59:32.764 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34837.log 2024-09-13T11:59:32.764 INFO:teuthology.orchestra.run.smithi100.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38372.log.gz 2024-09-13T11:59:32.764 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36435.log 2024-09-13T11:59:32.764 INFO:teuthology.orchestra.run.smithi100.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35604.log.gz 2024-09-13T11:59:32.764 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.34837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39170.log 2024-09-13T11:59:32.764 INFO:teuthology.orchestra.run.smithi100.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37377.log.gz 2024-09-13T11:59:32.764 INFO:teuthology.orchestra.run.smithi100.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34837.log.gz 2024-09-13T11:59:32.764 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35444.log/var/log/ceph/ceph-client.admin.36435.log: 2024-09-13T11:59:32.765 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39002.log 2024-09-13T11:59:32.765 INFO:teuthology.orchestra.run.smithi100.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.36435.log.gz 2024-09-13T11:59:32.765 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35018.log 2024-09-13T11:59:32.765 INFO:teuthology.orchestra.run.smithi100.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39170.log.gz 2024-09-13T11:59:32.765 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39002.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35444.log.gz 2024-09-13T11:59:32.765 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35445.log 2024-09-13T11:59:32.765 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39044.log 2024-09-13T11:59:32.765 INFO:teuthology.orchestra.run.smithi100.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39002.log.gz 2024-09-13T11:59:32.765 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35445.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-13T11:59:32.765 INFO:teuthology.orchestra.run.smithi100.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35018.log.gz 2024-09-13T11:59:32.766 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34702.log 2024-09-13T11:59:32.766 INFO:teuthology.orchestra.run.smithi100.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35445.log.gz 2024-09-13T11:59:32.766 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-osd.1.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39044.log.gz 2024-09-13T11:59:32.766 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38918.log 2024-09-13T11:59:32.766 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.34702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35525.log 2024-09-13T11:59:32.766 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37934.log 2024-09-13T11:59:32.766 INFO:teuthology.orchestra.run.smithi100.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34702.log.gz 2024-09-13T11:59:32.766 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38414.log 2024-09-13T11:59:32.766 INFO:teuthology.orchestra.run.smithi100.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38918.log.gz 2024-09-13T11:59:32.766 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35971.log 2024-09-13T11:59:32.767 INFO:teuthology.orchestra.run.smithi100.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35525.log.gz 2024-09-13T11:59:32.767 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38792.log 2024-09-13T11:59:32.767 INFO:teuthology.orchestra.run.smithi100.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37934.log.gz 2024-09-13T11:59:32.767 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37173.log 2024-09-13T11:59:32.767 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38792.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.38414.log.gz 2024-09-13T11:59:32.767 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35521.log 2024-09-13T11:59:32.767 INFO:teuthology.orchestra.run.smithi100.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35971.log.gz 2024-09-13T11:59:32.767 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36013.log 2024-09-13T11:59:32.767 INFO:teuthology.orchestra.run.smithi100.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38792.log.gz 2024-09-13T11:59:32.767 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32006.log 2024-09-13T11:59:32.768 INFO:teuthology.orchestra.run.smithi100.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37173.log.gz 2024-09-13T11:59:32.768 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36013.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35521.log.gz 2024-09-13T11:59:32.768 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36497.log 2024-09-13T11:59:32.768 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37510.log 2024-09-13T11:59:32.768 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.32006.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36013.log.gz 2024-09-13T11:59:32.768 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35803.log 2024-09-13T11:59:32.768 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37510.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32006.log.gz 2024-09-13T11:59:32.768 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37850.log 2024-09-13T11:59:32.768 INFO:teuthology.orchestra.run.smithi100.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36497.log.gz 2024-09-13T11:59:32.768 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37892.log 2024-09-13T11:59:32.769 INFO:teuthology.orchestra.run.smithi100.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37510.log.gz 2024-09-13T11:59:32.769 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37976.log 2024-09-13T11:59:32.769 INFO:teuthology.orchestra.run.smithi100.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35803.log.gz 2024-09-13T11:59:32.769 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34621.log 2024-09-13T11:59:32.769 INFO:teuthology.orchestra.run.smithi100.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37850.log.gz 2024-09-13T11:59:32.769 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38288.log 2024-09-13T11:59:32.769 INFO:teuthology.orchestra.run.smithi100.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37892.log.gz 2024-09-13T11:59:32.769 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.34621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31935.log 2024-09-13T11:59:32.769 INFO:teuthology.orchestra.run.smithi100.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37976.log.gz 2024-09-13T11:59:32.769 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39378.log 2024-09-13T11:59:32.770 INFO:teuthology.orchestra.run.smithi100.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34621.log.gz 2024-09-13T11:59:32.770 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.31935.log: gzip -5 --verbose 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38288.log.gz 2024-09-13T11:59:32.770 INFO:teuthology.orchestra.run.smithi100.stderr: -- /var/log/ceph/ceph-client.admin.38582.log 2024-09-13T11:59:32.770 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39378.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31935.log.gz 2024-09-13T11:59:32.770 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35272.log 2024-09-13T11:59:32.770 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38582.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39378.log.gz 2024-09-13T11:59:32.770 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-13T11:59:32.770 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35272.log: gzip 88.5% -5 --verbose -- /var/log/ceph/ceph-client.admin.35845.log 2024-09-13T11:59:32.770 INFO:teuthology.orchestra.run.smithi100.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38582.log.gz 2024-09-13T11:59:32.770 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph.audit.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35272.log.gz 2024-09-13T11:59:32.771 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39212.log 2024-09-13T11:59:32.771 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38018.log 2024-09-13T11:59:32.771 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39212.log: gzip -5 89.5% --verbose -- replaced with /var/log/ceph/ceph-client.admin.35845.log.gz -- 2024-09-13T11:59:32.771 INFO:teuthology.orchestra.run.smithi100.stderr: /var/log/ceph/ceph-client.admin.36141.log 2024-09-13T11:59:32.771 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38018.log: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-13T11:59:32.771 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36961.log 2024-09-13T11:59:32.771 INFO:teuthology.orchestra.run.smithi100.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39212.log.gz 2024-09-13T11:59:32.771 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36351.log 2024-09-13T11:59:32.771 INFO:teuthology.orchestra.run.smithi100.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38018.log.gz 2024-09-13T11:59:32.771 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35271.log 2024-09-13T11:59:32.772 INFO:teuthology.orchestra.run.smithi100.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.36141.log.gz 2024-09-13T11:59:32.772 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35270.log 2024-09-13T11:59:32.772 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35271.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36961.log.gz 2024-09-13T11:59:32.772 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35021.log 2024-09-13T11:59:32.772 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37003.log 2024-09-13T11:59:32.772 INFO:teuthology.orchestra.run.smithi100.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35271.log.gz 2024-09-13T11:59:32.772 INFO:teuthology.orchestra.run.smithi100.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.36351.log.gz 2024-09-13T11:59:32.772 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35273.log 2024-09-13T11:59:32.772 INFO:teuthology.orchestra.run.smithi100.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35270.log.gz 2024-09-13T11:59:32.773 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37003.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35021.log.gz 2024-09-13T11:59:32.773 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36623.log 2024-09-13T11:59:32.773 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37638.log 2024-09-13T11:59:32.773 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35273.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37003.log.gz 2024-09-13T11:59:32.773 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36057.log 2024-09-13T11:59:32.773 INFO:teuthology.orchestra.run.smithi100.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35273.log.gz 2024-09-13T11:59:32.773 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37722.log 2024-09-13T11:59:32.773 INFO:teuthology.orchestra.run.smithi100.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36623.log.gz 2024-09-13T11:59:32.773 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36309.log 2024-09-13T11:59:32.773 INFO:teuthology.orchestra.run.smithi100.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.37638.log.gz 2024-09-13T11:59:32.774 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log/var/log/ceph/ceph-client.admin.37722.log: 2024-09-13T11:59:32.774 INFO:teuthology.orchestra.run.smithi100.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36057.log.gz 2024-09-13T11:59:32.774 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34753.log 2024-09-13T11:59:32.774 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-mon.c.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37722.log.gz 2024-09-13T11:59:32.774 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- 88.6% /var/log/ceph/ceph-client.admin.37594.log 2024-09-13T11:59:32.774 INFO:teuthology.orchestra.run.smithi100.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36309.log.gz 2024-09-13T11:59:32.774 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37131.log 2024-09-13T11:59:32.774 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.34753.log: /var/log/ceph/ceph-client.admin.37594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37294.log 2024-09-13T11:59:32.775 INFO:teuthology.orchestra.run.smithi100.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34753.log.gz 2024-09-13T11:59:32.775 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34795.log 2024-09-13T11:59:32.775 INFO:teuthology.orchestra.run.smithi100.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.37594.log.gz 2024-09-13T11:59:32.775 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38750.log 2024-09-13T11:59:32.775 INFO:teuthology.orchestra.run.smithi100.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37131.log.gz 2024-09-13T11:59:32.775 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.34795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38060.log 2024-09-13T11:59:32.775 INFO:teuthology.orchestra.run.smithi100.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37294.log.gz 2024-09-13T11:59:32.775 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38750.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34795.log.gz 2024-09-13T11:59:32.775 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27336.log 2024-09-13T11:59:32.775 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38060.log: gzip 88.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.38750.log.gz 2024-09-13T11:59:32.776 INFO:teuthology.orchestra.run.smithi100.stderr: --verbose -- /var/log/ceph/ceph-client.admin.37045.log 2024-09-13T11:59:32.776 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph.tmp-client.admin.27336.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27336.log.gz 2024-09-13T11:59:32.776 INFO:teuthology.orchestra.run.smithi100.stderr: --verbose -- /var/log/ceph/ceph-client.admin.36835.log 2024-09-13T11:59:32.776 INFO:teuthology.orchestra.run.smithi100.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.38060.log.gz 2024-09-13T11:59:32.776 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31706.log 2024-09-13T11:59:32.776 INFO:teuthology.orchestra.run.smithi100.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37045.log.gz 2024-09-13T11:59:32.776 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36707.log 2024-09-13T11:59:32.776 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36581.log 2024-09-13T11:59:32.776 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.31706.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36835.log.gz 2024-09-13T11:59:32.777 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35020.log 2024-09-13T11:59:32.777 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36581.log: gzip 91.4% -- replaced with /var/log/ceph/ceph-client.admin.31706.log.gz 2024-09-13T11:59:32.777 INFO:teuthology.orchestra.run.smithi100.stderr: -5 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36707.log.gz 2024-09-13T11:59:32.777 INFO:teuthology.orchestra.run.smithi100.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38104.log 2024-09-13T11:59:32.777 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35020.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.36581.log.gz 2024-09-13T11:59:32.777 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-13T11:59:32.777 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36183.log 2024-09-13T11:59:32.777 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38104.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35020.log.gz 2024-09-13T11:59:32.777 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38960.log 2024-09-13T11:59:32.777 INFO:teuthology.orchestra.run.smithi100.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38104.log.gz 2024-09-13T11:59:32.778 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35677.log 2024-09-13T11:59:32.778 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36267.log 2024-09-13T11:59:32.778 INFO:teuthology.orchestra.run.smithi100.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36183.log.gz 2024-09-13T11:59:32.778 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37087.log 2024-09-13T11:59:32.778 INFO:teuthology.orchestra.run.smithi100.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38960.log.gz 2024-09-13T11:59:32.778 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35719.log 2024-09-13T11:59:32.778 INFO:teuthology.orchestra.run.smithi100.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35677.log.gz 2024-09-13T11:59:32.778 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36267.log: /var/log/ceph/ceph-client.admin.37087.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-13T11:59:32.778 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35719.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36267.log.gz 2024-09-13T11:59:32.778 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35266.log 2024-09-13T11:59:32.779 INFO:teuthology.orchestra.run.smithi100.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.37087.log.gz 2024-09-13T11:59:32.779 INFO:teuthology.orchestra.run.smithi100.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35719.log.gz 2024-09-13T11:59:32.779 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37680.log 2024-09-13T11:59:32.779 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37552.log 2024-09-13T11:59:32.779 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37680.log: gzip -5 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35266.log.gz 2024-09-13T11:59:32.779 INFO:teuthology.orchestra.run.smithi100.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34530.log 2024-09-13T11:59:32.779 INFO:teuthology.orchestra.run.smithi100.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37680.log.gz 2024-09-13T11:59:32.779 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39338.log 2024-09-13T11:59:32.779 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.34530.log: /var/log/ceph/ceph-client.admin.37552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36099.log 2024-09-13T11:59:32.779 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39338.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34530.log.gz 2024-09-13T11:59:32.780 INFO:teuthology.orchestra.run.smithi100.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.37552.log.gz 2024-09-13T11:59:32.780 INFO:teuthology.orchestra.run.smithi100.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39338.log.gz 2024-09-13T11:59:32.780 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38708.log 2024-09-13T11:59:32.780 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35009.log 2024-09-13T11:59:32.780 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38666.log 2024-09-13T11:59:32.780 INFO:teuthology.orchestra.run.smithi100.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.36099.log.gz 2024-09-13T11:59:32.780 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35009.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38708.log.gz 2024-09-13T11:59:32.780 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-13T11:59:32.781 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38456.log 2024-09-13T11:59:32.781 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35009.log.gz 2024-09-13T11:59:32.781 INFO:teuthology.orchestra.run.smithi100.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38666.log.gz 2024-09-13T11:59:32.781 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39128.log 2024-09-13T11:59:32.781 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35605.log 2024-09-13T11:59:32.781 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39128.log: /var/log/ceph/ceph-client.admin.38456.log: 88.8% 90.0% -- replaced with /var/log/ceph/ceph-client.admin.38456.log.gz 2024-09-13T11:59:32.781 INFO:teuthology.orchestra.run.smithi100.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39128.log.gz 2024-09-13T11:59:32.781 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38834.log 2024-09-13T11:59:32.781 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36749.log 2024-09-13T11:59:32.782 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35605.log: /var/log/ceph/ceph-client.admin.38834.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35605.log.gz 2024-09-13T11:59:32.782 INFO:teuthology.orchestra.run.smithi100.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38834.log.gz 2024-09-13T11:59:32.782 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34879.log 2024-09-13T11:59:32.782 INFO:teuthology.orchestra.run.smithi100.stderr: 91.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-13T11:59:32.782 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38540.log 2024-09-13T11:59:32.782 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.34879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38876.log 2024-09-13T11:59:32.782 INFO:teuthology.orchestra.run.smithi100.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36749.log.gz 2024-09-13T11:59:32.782 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38540.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34879.log.gz 2024-09-13T11:59:32.782 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36793.log 2024-09-13T11:59:32.782 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39296.log 2024-09-13T11:59:32.782 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36793.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38540.log.gz 2024-09-13T11:59:32.783 INFO:teuthology.orchestra.run.smithi100.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38876.log.gz 2024-09-13T11:59:32.783 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36665.log 2024-09-13T11:59:32.783 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 90.2% -- replaced with /var/log/ceph/ceph-client.admin.36793.log.gz 2024-09-13T11:59:32.783 INFO:teuthology.orchestra.run.smithi100.stderr: --verbose -- /var/log/ceph/ceph-client.admin.36225.log 2024-09-13T11:59:32.783 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36665.log: /var/log/ceph/ceph-client.admin.39296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38330.log 2024-09-13T11:59:32.783 INFO:teuthology.orchestra.run.smithi100.stderr: 89.8% 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39296.log.gz 2024-09-13T11:59:32.783 INFO:teuthology.orchestra.run.smithi100.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36665.log.gz 2024-09-13T11:59:32.783 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35887.log 2024-09-13T11:59:32.784 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38330.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36225.log.gz 2024-09-13T11:59:32.784 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38498.log 2024-09-13T11:59:32.784 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35887.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38330.log.gz 2024-09-13T11:59:32.784 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36877.log 2024-09-13T11:59:32.784 INFO:teuthology.orchestra.run.smithi100.stderr: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39254.log 2024-09-13T11:59:32.784 INFO:teuthology.orchestra.run.smithi100.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35887.log.gz 2024-09-13T11:59:32.784 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38498.log: /var/log/ceph/ceph-client.admin.36877.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36877.log.gz 2024-09-13T11:59:32.784 INFO:teuthology.orchestra.run.smithi100.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.38498.log.gz 2024-09-13T11:59:32.784 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35269.log 2024-09-13T11:59:32.784 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38624.log 2024-09-13T11:59:32.785 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35269.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39254.log.gz 2024-09-13T11:59:32.785 INFO:teuthology.orchestra.run.smithi100.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39086.log 2024-09-13T11:59:32.785 INFO:teuthology.orchestra.run.smithi100.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35269.log.gz 2024-09-13T11:59:32.785 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35929.log 2024-09-13T11:59:32.785 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38624.log: /var/log/ceph/ceph-client.admin.39086.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38624.log.gz 2024-09-13T11:59:32.785 INFO:teuthology.orchestra.run.smithi100.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39086.log.gz 2024-09-13T11:59:32.785 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36393.log 2024-09-13T11:59:32.785 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36919.log 2024-09-13T11:59:32.785 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36393.log: gzip -5 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35929.log.gz 2024-09-13T11:59:32.786 INFO:teuthology.orchestra.run.smithi100.stderr: --verbose -- /var/log/ceph/ceph-client.admin.35017.log 2024-09-13T11:59:32.786 INFO:teuthology.orchestra.run.smithi100.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36393.log.gz 2024-09-13T11:59:32.786 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36919.log: /var/log/ceph/ceph-client.admin.35017.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36919.log.gz 2024-09-13T11:59:32.786 INFO:teuthology.orchestra.run.smithi100.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35017.log.gz 2024-09-13T11:59:32.875 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37398.log 2024-09-13T11:59:32.875 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38050.log 2024-09-13T11:59:32.875 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35766.log 2024-09-13T11:59:32.875 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37398.log: /var/log/ceph/ceph-client.admin.38050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39253.log 2024-09-13T11:59:32.875 INFO:teuthology.orchestra.run.smithi122.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37398.log.gz 2024-09-13T11:59:32.875 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38324.log 2024-09-13T11:59:32.875 INFO:teuthology.orchestra.run.smithi122.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38050.log.gz 2024-09-13T11:59:32.875 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34874.log 2024-09-13T11:59:32.875 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35766.log.gz 2024-09-13T11:59:32.875 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39357.log 2024-09-13T11:59:32.876 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.34874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38944.log 2024-09-13T11:59:32.876 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39357.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39253.log.gz 2024-09-13T11:59:32.876 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36403.log 2024-09-13T11:59:32.876 INFO:teuthology.orchestra.run.smithi122.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34874.log.gz 2024-09-13T11:59:32.876 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38324.log.gz 2024-09-13T11:59:32.876 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38944.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39357.log.gz 2024-09-13T11:59:32.876 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36574.log 2024-09-13T11:59:32.876 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37706.log 2024-09-13T11:59:32.876 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35444.log 2024-09-13T11:59:32.876 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37706.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38944.log.gz 2024-09-13T11:59:32.876 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35339.log 2024-09-13T11:59:32.876 INFO:teuthology.orchestra.run.smithi122.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36574.log.gz 2024-09-13T11:59:32.877 INFO:teuthology.orchestra.run.smithi122.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.36403.log.gz 2024-09-13T11:59:32.877 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39150.log 2024-09-13T11:59:32.877 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37913.log 2024-09-13T11:59:32.877 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37706.log.gz 2024-09-13T11:59:32.877 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37122.log 2024-09-13T11:59:32.877 INFO:teuthology.orchestra.run.smithi122.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35444.log.gz 2024-09-13T11:59:32.877 INFO:teuthology.orchestra.run.smithi122.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.35339.log.gz 2024-09-13T11:59:32.877 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35231.log 2024-09-13T11:59:32.877 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35018.log 2024-09-13T11:59:32.877 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37913.log.gz 2024-09-13T11:59:32.877 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35445.log 2024-09-13T11:59:32.877 INFO:teuthology.orchestra.run.smithi122.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.39150.log.gz 2024-09-13T11:59:32.878 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35408.log 2024-09-13T11:59:32.878 INFO:teuthology.orchestra.run.smithi122.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35231.log.gz 2024-09-13T11:59:32.878 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35445.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37122.log.gz 2024-09-13T11:59:32.878 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38566.log 2024-09-13T11:59:32.878 INFO:teuthology.orchestra.run.smithi122.stderr: 90.8%/var/log/ceph/ceph-client.admin.35408.log: -- replaced with /var/log/ceph/ceph-client.admin.35018.log.gz 2024-09-13T11:59:32.878 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35445.log.gz 2024-09-13T11:59:32.878 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37534.log 2024-09-13T11:59:32.878 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38668.log 2024-09-13T11:59:32.878 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36087.log 2024-09-13T11:59:32.878 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35408.log.gz 2024-09-13T11:59:32.878 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38668.log: 90.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.38566.log.gz 2024-09-13T11:59:32.878 INFO:teuthology.orchestra.run.smithi122.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36158.log 2024-09-13T11:59:32.879 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38668.log.gz 2024-09-13T11:59:32.879 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39460.log 2024-09-13T11:59:32.879 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38840.log 2024-09-13T11:59:32.879 INFO:teuthology.orchestra.run.smithi122.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.37534.log.gz 2024-09-13T11:59:32.879 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36087.log.gz 2024-09-13T11:59:32.879 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37604.log 2024-09-13T11:59:32.879 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36158.log.gz 2024-09-13T11:59:32.879 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37226.log 2024-09-13T11:59:32.879 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39460.log.gz 2024-09-13T11:59:32.879 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39392.log 2024-09-13T11:59:32.879 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35302.log 2024-09-13T11:59:32.879 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37604.log.gz 2024-09-13T11:59:32.879 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38840.log.gz 2024-09-13T11:59:32.879 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36300.log 2024-09-13T11:59:32.880 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37226.log.gz 2024-09-13T11:59:32.880 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38255.log 2024-09-13T11:59:32.880 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36300.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.39392.log.gzgzip 2024-09-13T11:59:32.880 INFO:teuthology.orchestra.run.smithi122.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38051.log 2024-09-13T11:59:32.880 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35302.log.gz 2024-09-13T11:59:32.880 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37431.log 2024-09-13T11:59:32.880 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38772.log 2024-09-13T11:59:32.880 INFO:teuthology.orchestra.run.smithi122.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.36300.log.gz 2024-09-13T11:59:32.880 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38256.log 2024-09-13T11:59:32.880 INFO:teuthology.orchestra.run.smithi122.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38255.log.gz 2024-09-13T11:59:32.880 INFO:teuthology.orchestra.run.smithi122.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.38051.log.gz 2024-09-13T11:59:32.880 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37330.log 2024-09-13T11:59:32.881 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36050.log 2024-09-13T11:59:32.881 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38256.log: /var/log/ceph/ceph-client.admin.37330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37328.log 2024-09-13T11:59:32.881 INFO:teuthology.orchestra.run.smithi122.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.38772.log.gz 2024-09-13T11:59:32.881 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37431.log.gz 2024-09-13T11:59:32.881 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36050.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38256.log.gz 2024-09-13T11:59:32.881 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35943.log 2024-09-13T11:59:32.881 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36157.log 2024-09-13T11:59:32.881 INFO:teuthology.orchestra.run.smithi122.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.37330.log.gz 2024-09-13T11:59:32.881 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39459.log 2024-09-13T11:59:32.881 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36050.log.gz 2024-09-13T11:59:32.881 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36882.log 2024-09-13T11:59:32.881 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37328.log.gz 2024-09-13T11:59:32.881 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35553.log 2024-09-13T11:59:32.882 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39459.log: /var/log/ceph/ceph-client.admin.36882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35765.log 2024-09-13T11:59:32.882 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36157.log.gz 2024-09-13T11:59:32.882 INFO:teuthology.orchestra.run.smithi122.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35943.log.gz 2024-09-13T11:59:32.882 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38979.log 2024-09-13T11:59:32.882 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36505.log 2024-09-13T11:59:32.882 INFO:teuthology.orchestra.run.smithi122.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39459.log.gz 2024-09-13T11:59:32.882 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36882.log.gz 2024-09-13T11:59:32.882 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38152.log 2024-09-13T11:59:32.882 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38979.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.35553.log.gz 2024-09-13T11:59:32.882 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38634.log 2024-09-13T11:59:32.882 INFO:teuthology.orchestra.run.smithi122.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38979.log.gz 2024-09-13T11:59:32.882 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38152.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36505.log.gz 2024-09-13T11:59:32.882 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35979.log 2024-09-13T11:59:32.883 INFO:teuthology.orchestra.run.smithi122.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.35765.log.gz 2024-09-13T11:59:32.883 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35944.log 2024-09-13T11:59:32.883 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36506.log 2024-09-13T11:59:32.883 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38634.log.gz 2024-09-13T11:59:32.883 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38152.log.gz 2024-09-13T11:59:32.883 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38463.log 2024-09-13T11:59:32.883 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34554.log 2024-09-13T11:59:32.883 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35944.log.gz 2024-09-13T11:59:32.883 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38463.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35979.log.gz 2024-09-13T11:59:32.883 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37225.log 2024-09-13T11:59:32.883 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.34554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31498.log 2024-09-13T11:59:32.883 INFO:teuthology.orchestra.run.smithi122.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.36506.log.gz 2024-09-13T11:59:32.884 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37225.log: gzip 90.6% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.38463.log.gz 2024-09-13T11:59:32.884 INFO:teuthology.orchestra.run.smithi122.stderr: -- /var/log/ceph/ceph-client.admin.36401.log 2024-09-13T11:59:32.884 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34554.log.gz 2024-09-13T11:59:32.884 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.31498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36086.log 2024-09-13T11:59:32.884 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38667.log 2024-09-13T11:59:32.884 INFO:teuthology.orchestra.run.smithi122.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31498.log.gz 2024-09-13T11:59:32.884 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37225.log.gz 2024-09-13T11:59:32.884 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39289.log 2024-09-13T11:59:32.884 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39287.log 2024-09-13T11:59:32.884 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38667.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36401.log.gz 2024-09-13T11:59:32.884 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38462.log 2024-09-13T11:59:32.884 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39289.log: /var/log/ceph/ceph-client.admin.39287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38016.log 2024-09-13T11:59:32.884 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36086.log.gz 2024-09-13T11:59:32.885 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38015.log 2024-09-13T11:59:32.885 INFO:teuthology.orchestra.run.smithi122.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.39289.log.gz 2024-09-13T11:59:32.885 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38667.log.gz 2024-09-13T11:59:32.885 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34589.log 2024-09-13T11:59:32.885 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38016.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38462.log.gz 2024-09-13T11:59:32.885 INFO:teuthology.orchestra.run.smithi122.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39287.log.gz 2024-09-13T11:59:32.885 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36193.log 2024-09-13T11:59:32.885 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38016.log.gz 2024-09-13T11:59:32.885 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.34589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36367.log 2024-09-13T11:59:32.885 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37535.log 2024-09-13T11:59:32.886 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36367.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38015.log.gz 2024-09-13T11:59:32.886 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36368.log 2024-09-13T11:59:32.886 INFO:teuthology.orchestra.run.smithi122.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.34589.log.gz 2024-09-13T11:59:32.886 INFO:teuthology.orchestra.run.smithi122.stderr:gzip/var/log/ceph/ceph-client.admin.37535.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.37844.log 2024-09-13T11:59:32.886 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36368.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36193.log.gz 2024-09-13T11:59:32.886 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-13T11:59:32.886 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37535.log.gz 2024-09-13T11:59:32.886 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36367.log.gz 2024-09-13T11:59:32.886 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35622.log 2024-09-13T11:59:32.886 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37844.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36368.log.gz 2024-09-13T11:59:32.886 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36677.log 2024-09-13T11:59:32.886 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37844.log.gz 2024-09-13T11:59:32.886 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36780.log 2024-09-13T11:59:32.887 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37810.log 2024-09-13T11:59:32.887 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36780.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36677.log.gz 2024-09-13T11:59:32.887 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35123.log 2024-09-13T11:59:32.887 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35622.log.gz 2024-09-13T11:59:32.887 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36780.log.gz 2024-09-13T11:59:32.887 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37810.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-13T11:59:32.887 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39083.log 2024-09-13T11:59:32.887 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37810.log.gz 2024-09-13T11:59:32.887 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39596.log 2024-09-13T11:59:32.887 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38565.log 2024-09-13T11:59:32.887 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-13T11:59:32.888 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38359.log 2024-09-13T11:59:32.888 INFO:teuthology.orchestra.run.smithi122.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35123.log.gz 2024-09-13T11:59:32.888 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37432.log 2024-09-13T11:59:32.888 INFO:teuthology.orchestra.run.smithi122.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.39083.log.gz 2024-09-13T11:59:32.888 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35873.log 2024-09-13T11:59:32.888 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38359.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38565.log.gz 2024-09-13T11:59:32.888 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37809.log 2024-09-13T11:59:32.888 INFO:teuthology.orchestra.run.smithi122.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.39596.log.gz 2024-09-13T11:59:32.888 INFO:teuthology.orchestra.run.smithi122.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38359.log.gz 2024-09-13T11:59:32.888 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39082.log 2024-09-13T11:59:32.889 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37432.log.gz 2024-09-13T11:59:32.889 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35873.log.gz 2024-09-13T11:59:32.892 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36504.log 2024-09-13T11:59:32.892 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34696.log 2024-09-13T11:59:32.892 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35195.log 2024-09-13T11:59:32.892 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39082.log.gz 2024-09-13T11:59:32.892 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37809.log.gz 2024-09-13T11:59:32.892 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.34696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34910.log 2024-09-13T11:59:32.892 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38978.log 2024-09-13T11:59:32.893 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34696.log.gz 2024-09-13T11:59:32.893 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35195.log: /var/log/ceph/ceph-client.admin.34910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34659.log 2024-09-13T11:59:32.893 INFO:teuthology.orchestra.run.smithi122.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.36504.log.gz 2024-09-13T11:59:32.893 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38978.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.35195.log.gz 2024-09-13T11:59:32.893 INFO:teuthology.orchestra.run.smithi122.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37740.log 2024-09-13T11:59:32.893 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34910.log.gz 2024-09-13T11:59:32.893 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37603.log 2024-09-13T11:59:32.893 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.34659.log: /var/log/ceph/ceph-client.admin.37740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34660.log 2024-09-13T11:59:32.893 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38978.log.gz 2024-09-13T11:59:32.893 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37707.log 2024-09-13T11:59:32.894 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34659.log.gz 2024-09-13T11:59:32.894 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.34660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31554.log 2024-09-13T11:59:32.894 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37707.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37740.log.gz 2024-09-13T11:59:32.894 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38531.log 2024-09-13T11:59:32.894 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34660.log.gz 2024-09-13T11:59:32.894 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37603.log.gz 2024-09-13T11:59:32.894 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.31554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36051.log 2024-09-13T11:59:32.894 INFO:teuthology.orchestra.run.smithi122.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37707.log.gz 2024-09-13T11:59:32.894 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38531.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31554.log.gz 2024-09-13T11:59:32.894 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36985.log 2024-09-13T11:59:32.894 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36609.log 2024-09-13T11:59:32.895 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36051.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38531.log.gz 2024-09-13T11:59:32.895 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39151.log 2024-09-13T11:59:32.895 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36609.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36051.log.gz 2024-09-13T11:59:32.895 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36299.log 2024-09-13T11:59:32.895 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38770.log 2024-09-13T11:59:32.895 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39151.log: /var/log/ceph/ceph-client.admin.36299.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.36609.log.gz 2024-09-13T11:59:32.895 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35124.log 2024-09-13T11:59:32.895 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36985.log.gz 2024-09-13T11:59:32.895 INFO:teuthology.orchestra.run.smithi122.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39151.log.gz 2024-09-13T11:59:32.895 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37946.log 2024-09-13T11:59:32.895 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38770.log: /var/log/ceph/ceph-client.admin.35124.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36299.log.gz 2024-09-13T11:59:32.895 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35125.log 2024-09-13T11:59:32.896 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37638.log 2024-09-13T11:59:32.896 INFO:teuthology.orchestra.run.smithi122.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35124.log.gz 2024-09-13T11:59:32.896 INFO:teuthology.orchestra.run.smithi122.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38770.log.gz 2024-09-13T11:59:32.896 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39391.log 2024-09-13T11:59:32.896 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38669.log 2024-09-13T11:59:32.896 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37021.log 2024-09-13T11:59:32.896 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37946.log.gz 2024-09-13T11:59:32.896 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37638.log.gz 2024-09-13T11:59:32.896 INFO:teuthology.orchestra.run.smithi122.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.35125.log.gz 2024-09-13T11:59:32.896 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37191.log 2024-09-13T11:59:32.897 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39391.log.gz 2024-09-13T11:59:32.897 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37294.log 2024-09-13T11:59:32.897 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36194.log 2024-09-13T11:59:32.897 INFO:teuthology.orchestra.run.smithi122.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.38669.log.gz 2024-09-13T11:59:32.897 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36986.log 2024-09-13T11:59:32.897 INFO:teuthology.orchestra.run.smithi122.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.37021.log.gz 2024-09-13T11:59:32.897 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37294.log: /var/log/ceph/ceph-client.admin.36194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39390.log 2024-09-13T11:59:32.897 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37191.log.gz 2024-09-13T11:59:32.897 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35836.log 2024-09-13T11:59:32.897 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36194.log.gz 2024-09-13T11:59:32.898 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35872.log 2024-09-13T11:59:32.898 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36986.log.gz 2024-09-13T11:59:32.898 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39390.log: /var/log/ceph/ceph-client.admin.35836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39047.log 2024-09-13T11:59:32.898 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37294.log.gz 2024-09-13T11:59:32.898 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38633.log 2024-09-13T11:59:32.898 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39390.log.gz 2024-09-13T11:59:32.898 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37329.log 2024-09-13T11:59:32.898 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34804.log 2024-09-13T11:59:32.898 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38633.log: 89.2% 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35872.log.gz 2024-09-13T11:59:32.898 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35836.log.gz 2024-09-13T11:59:32.898 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35232.log 2024-09-13T11:59:32.898 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35729.log 2024-09-13T11:59:32.899 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.34804.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.39047.log.gz 2024-09-13T11:59:32.899 INFO:teuthology.orchestra.run.smithi122.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37329.log.gz 2024-09-13T11:59:32.899 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34911.log 2024-09-13T11:59:32.899 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38633.log.gz 2024-09-13T11:59:32.899 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38325.log 2024-09-13T11:59:32.899 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35729.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.34804.log.gz 2024-09-13T11:59:32.899 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.34911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36676.log 2024-09-13T11:59:32.899 INFO:teuthology.orchestra.run.smithi122.stderr:gzip 90.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.35232.log.gz --verbose 2024-09-13T11:59:32.899 INFO:teuthology.orchestra.run.smithi122.stderr: -- /var/log/ceph/ceph-client.admin.38358.log 2024-09-13T11:59:32.899 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38325.log: /var/log/ceph/ceph-client.admin.36676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36195.log 2024-09-13T11:59:32.900 INFO:teuthology.orchestra.run.smithi122.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35729.log.gz 2024-09-13T11:59:32.900 INFO:teuthology.orchestra.run.smithi122.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.34911.log.gz 2024-09-13T11:59:32.900 INFO:teuthology.orchestra.run.smithi122.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38325.log.gz 2024-09-13T11:59:32.900 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37912.log 2024-09-13T11:59:32.900 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34909.log 2024-09-13T11:59:32.900 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36195.log: /var/log/ceph/ceph-client.admin.37912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37192.log 2024-09-13T11:59:32.900 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36676.log.gz 2024-09-13T11:59:32.900 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.34909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37501.log 2024-09-13T11:59:32.900 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38358.log.gz 2024-09-13T11:59:32.900 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35874.log 2024-09-13T11:59:32.901 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37192.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.36195.log.gz 2024-09-13T11:59:32.901 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39597.log 2024-09-13T11:59:32.901 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34909.log.gz 2024-09-13T11:59:32.901 INFO:teuthology.orchestra.run.smithi122.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.37912.log.gz 2024-09-13T11:59:32.901 INFO:teuthology.orchestra.run.smithi122.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37192.log.gz 2024-09-13T11:59:32.901 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38841.log 2024-09-13T11:59:32.901 INFO:teuthology.orchestra.run.smithi122.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37501.log.gz 2024-09-13T11:59:32.901 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35837.log 2024-09-13T11:59:32.901 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38841.log: gzip -5 --verbose 90.3% -- -- replaced with /var/log/ceph/ceph-client.admin.35874.log.gz /var/log/ceph/ceph-client.admin.38736.log 2024-09-13T11:59:32.901 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:59:32.901 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39597.log.gz 2024-09-13T11:59:32.902 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36471.log 2024-09-13T11:59:32.902 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35837.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38841.log.gz 2024-09-13T11:59:32.902 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37500.log 2024-09-13T11:59:32.902 INFO:teuthology.orchestra.run.smithi122.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35837.log.gz 2024-09-13T11:59:32.902 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36265.log 2024-09-13T11:59:32.902 INFO:teuthology.orchestra.run.smithi122.stderr:gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36471.log.gz -5 2024-09-13T11:59:32.902 INFO:teuthology.orchestra.run.smithi122.stderr: --verbose -- /var/log/ceph/ceph-client.admin.37639.log 2024-09-13T11:59:32.902 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37500.log: /var/log/ceph/ceph-client.admin.36265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36711.log 2024-09-13T11:59:32.902 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37639.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38736.log.gz 2024-09-13T11:59:32.902 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35730.log 2024-09-13T11:59:32.903 INFO:teuthology.orchestra.run.smithi122.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36265.log.gz 2024-09-13T11:59:32.903 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36711.log: gzip -5 --verbose 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37500.log.gz 2024-09-13T11:59:32.903 INFO:teuthology.orchestra.run.smithi122.stderr: -- /var/log/ceph/ceph-client.admin.36918.log 2024-09-13T11:59:32.903 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35730.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36711.log.gz 2024-09-13T11:59:32.903 INFO:teuthology.orchestra.run.smithi122.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.37639.log.gz 2024-09-13T11:59:32.903 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38771.log 2024-09-13T11:59:32.903 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36402.log 2024-09-13T11:59:32.903 INFO:teuthology.orchestra.run.smithi122.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35730.log.gz 2024-09-13T11:59:32.903 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31674.log 2024-09-13T11:59:32.903 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36712.log 2024-09-13T11:59:32.904 INFO:teuthology.orchestra.run.smithi122.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.36918.log.gz 2024-09-13T11:59:32.904 INFO:teuthology.orchestra.run.smithi122.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38771.log.gz 2024-09-13T11:59:32.904 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36402.log.gz 2024-09-13T11:59:32.904 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39081.log 2024-09-13T11:59:32.904 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.31674.log: /var/log/ceph/ceph-client.admin.36712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37843.log 2024-09-13T11:59:32.904 INFO:teuthology.orchestra.run.smithi122.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.31674.log.gz 2024-09-13T11:59:32.904 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38945.log 2024-09-13T11:59:32.904 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35623.log 2024-09-13T11:59:32.904 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37843.log: /var/log/ceph/ceph-client.admin.38945.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.36712.log.gz 2024-09-13T11:59:32.904 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39186.log 2024-09-13T11:59:32.905 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39081.log.gz 2024-09-13T11:59:32.905 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38119.log 2024-09-13T11:59:32.905 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38945.log.gz 2024-09-13T11:59:32.905 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35338.log 2024-09-13T11:59:32.905 INFO:teuthology.orchestra.run.smithi122.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37843.log.gz 2024-09-13T11:59:32.905 INFO:teuthology.orchestra.run.smithi122.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35623.log.gz 2024-09-13T11:59:32.905 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35551.log 2024-09-13T11:59:32.905 INFO:teuthology.orchestra.run.smithi122.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.39186.log.gz 2024-09-13T11:59:32.905 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31458.log 2024-09-13T11:59:32.905 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38119.log.gz 2024-09-13T11:59:32.905 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37089.log 2024-09-13T11:59:32.906 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35551.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35338.log.gz 2024-09-13T11:59:32.906 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.31458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38980.log 2024-09-13T11:59:32.906 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36917.log 2024-09-13T11:59:32.906 INFO:teuthology.orchestra.run.smithi122.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31458.log.gz 2024-09-13T11:59:32.906 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38980.log: gzip -5 --verbose -- 89.4% /var/log/ceph/ceph-client.admin.35659.log -- replaced with /var/log/ceph/ceph-client.admin.35551.log.gz 2024-09-13T11:59:32.906 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:59:32.906 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37089.log.gz 2024-09-13T11:59:32.906 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39493.log 2024-09-13T11:59:32.906 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36917.log: /var/log/ceph/ceph-client.admin.35659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38461.log 2024-09-13T11:59:32.906 INFO:teuthology.orchestra.run.smithi122.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.38980.log.gz 2024-09-13T11:59:32.906 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36917.log.gz 2024-09-13T11:59:32.907 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36815.log 2024-09-13T11:59:32.907 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35659.log.gz 2024-09-13T11:59:32.907 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34767.log 2024-09-13T11:59:32.907 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36088.log 2024-09-13T11:59:32.907 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39493.log.gz 2024-09-13T11:59:32.907 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.34767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39288.log 2024-09-13T11:59:32.907 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36088.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38461.log.gz 2024-09-13T11:59:32.907 INFO:teuthology.orchestra.run.smithi122.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.36815.log.gz 2024-09-13T11:59:32.907 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38428.log 2024-09-13T11:59:32.907 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34767.log.gz 2024-09-13T11:59:32.907 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35088.log 2024-09-13T11:59:32.907 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35515.log 2024-09-13T11:59:32.907 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38428.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39288.log.gz 2024-09-13T11:59:32.908 INFO:teuthology.orchestra.run.smithi122.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.36088.log.gz 2024-09-13T11:59:32.908 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37227.log 2024-09-13T11:59:32.908 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38428.log.gz 2024-09-13T11:59:32.908 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34766.log 2024-09-13T11:59:32.908 INFO:teuthology.orchestra.run.smithi122.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35088.log.gz 2024-09-13T11:59:32.908 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37227.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-13T11:59:32.908 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.34766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38049.log 2024-09-13T11:59:32.908 INFO:teuthology.orchestra.run.smithi122.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35515.log.gz 2024-09-13T11:59:32.908 INFO:teuthology.orchestra.run.smithi122.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.37227.log.gz 2024-09-13T11:59:32.908 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36264.log 2024-09-13T11:59:32.908 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-13T11:59:32.908 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38049.log: /var/log/ceph/ceph-client.admin.36264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35660.log 2024-09-13T11:59:32.908 INFO:teuthology.orchestra.run.smithi122.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.34766.log.gz 2024-09-13T11:59:32.909 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34980.log 2024-09-13T11:59:32.909 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35660.log: 89.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34587.log 2024-09-13T11:59:32.909 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38049.log.gz 2024-09-13T11:59:32.909 INFO:teuthology.orchestra.run.smithi122.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.36264.log.gz 2024-09-13T11:59:32.909 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.34980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36916.log 2024-09-13T11:59:32.909 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39356.log 2024-09-13T11:59:32.909 INFO:teuthology.orchestra.run.smithi122.stderr: 90.6%/var/log/ceph/ceph-client.admin.34587.log: -- replaced with /var/log/ceph/ceph-client.admin.35660.log.gz 2024-09-13T11:59:32.909 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38874.log 2024-09-13T11:59:32.909 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39356.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34980.log.gz 2024-09-13T11:59:32.909 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38737.log 2024-09-13T11:59:32.909 INFO:teuthology.orchestra.run.smithi122.stderr: 86.9%/var/log/ceph/ceph-client.admin.38874.log: -- replaced with /var/log/ceph/ceph.log.gz 2024-09-13T11:59:32.909 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36779.log 2024-09-13T11:59:32.910 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36916.log.gz 2024-09-13T11:59:32.910 INFO:teuthology.orchestra.run.smithi122.stderr: 89.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35301.log 2024-09-13T11:59:32.910 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34587.log.gz 2024-09-13T11:59:32.910 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38737.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.39356.log.gz 2024-09-13T11:59:32.910 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34981.log 2024-09-13T11:59:32.910 INFO:teuthology.orchestra.run.smithi122.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38874.log.gz 2024-09-13T11:59:32.910 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35301.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38737.log.gz 2024-09-13T11:59:32.910 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-13T11:59:32.910 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.34981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38153.log 2024-09-13T11:59:32.910 INFO:teuthology.orchestra.run.smithi122.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34981.log.gz 2024-09-13T11:59:32.910 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-mgr.x.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36779.log.gz 2024-09-13T11:59:32.910 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36814.log 2024-09-13T11:59:32.910 INFO:teuthology.orchestra.run.smithi122.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35301.log.gz 2024-09-13T11:59:32.910 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37020.log 2024-09-13T11:59:32.911 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34802.log 2024-09-13T11:59:32.911 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38153.log.gz 2024-09-13T11:59:32.911 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39048.log 2024-09-13T11:59:32.911 INFO:teuthology.orchestra.run.smithi122.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36814.log.gz 2024-09-13T11:59:32.911 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.34802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36573.log 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37020.log.gz 2024-09-13T11:59:32.911 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:59:32.911 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35337.log 2024-09-13T11:59:32.911 INFO:teuthology.orchestra.run.smithi122.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.34802.log.gz 2024-09-13T11:59:32.911 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39494.log 2024-09-13T11:59:32.911 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39048.log.gz 2024-09-13T11:59:32.911 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34695.log 2024-09-13T11:59:32.911 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37088.log 2024-09-13T11:59:32.912 INFO:teuthology.orchestra.run.smithi122.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36573.log.gz 2024-09-13T11:59:32.912 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.34695.log: gzip -5 88.2% --verbose -- /var/log/ceph/ceph-client.admin.38154.log 2024-09-13T11:59:32.912 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39494.log.gz 2024-09-13T11:59:32.912 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35337.log.gz 2024-09-13T11:59:32.912 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38257.log 2024-09-13T11:59:32.912 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5/var/log/ceph/ceph-client.admin.38154.log: --verbose -- /var/log/ceph/ceph-client.admin.35658.log 2024-09-13T11:59:32.912 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38257.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34695.log.gz 2024-09-13T11:59:32.912 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37123.log 2024-09-13T11:59:32.912 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37088.log.gz 2024-09-13T11:59:32.912 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37397.log 2024-09-13T11:59:32.912 INFO:teuthology.orchestra.run.smithi122.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.38154.log.gz 2024-09-13T11:59:32.912 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35658.log: /var/log/ceph/ceph-client.admin.37123.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.38257.log.gz 2024-09-13T11:59:32.912 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36710.log 2024-09-13T11:59:32.912 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37123.log.gz 2024-09-13T11:59:32.913 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37397.log: gzip -5 --verbose 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35658.log.gz 2024-09-13T11:59:32.913 INFO:teuthology.orchestra.run.smithi122.stderr: -- /var/log/ceph/ceph-client.admin.37536.log 2024-09-13T11:59:32.913 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34697.log 2024-09-13T11:59:32.913 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37536.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37397.log.gz 2024-09-13T11:59:32.913 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37433.log 2024-09-13T11:59:32.913 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35516.log 2024-09-13T11:59:32.913 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36710.log.gz/var/log/ceph/ceph-client.admin.34697.log: 2024-09-13T11:59:32.913 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38530.log 2024-09-13T11:59:32.913 INFO:teuthology.orchestra.run.smithi122.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.37536.log.gz 2024-09-13T11:59:32.913 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.35516.log: -- /var/log/ceph/ceph-client.admin.38875.log 2024-09-13T11:59:32.913 INFO:teuthology.orchestra.run.smithi122.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.34697.log.gz 2024-09-13T11:59:32.913 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38530.log: 90.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35409.log 2024-09-13T11:59:32.914 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37433.log.gz 2024-09-13T11:59:32.914 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35516.log.gz 2024-09-13T11:59:32.914 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38876.log 2024-09-13T11:59:32.914 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5/var/log/ceph/ceph-client.admin.35409.log: --verbose -- /var/log/ceph/ceph-client.admin.35016.log 2024-09-13T11:59:32.914 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38875.log.gz 2024-09-13T11:59:32.914 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38876.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38530.log.gz 2024-09-13T11:59:32.914 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37741.log 2024-09-13T11:59:32.914 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35409.log.gz 2024-09-13T11:59:32.914 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39185.log 2024-09-13T11:59:32.914 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37741.log: gzip 90.3% -- replaced with /var/log/ceph/ceph-client.admin.38876.log.gz 2024-09-13T11:59:32.914 INFO:teuthology.orchestra.run.smithi122.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36883.log 2024-09-13T11:59:32.914 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38118.log 2024-09-13T11:59:32.915 INFO:teuthology.orchestra.run.smithi122.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35016.log.gz 2024-09-13T11:59:32.915 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37741.log.gz 2024-09-13T11:59:32.915 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36883.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39185.log.gz 2024-09-13T11:59:32.915 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39598.log 2024-09-13T11:59:32.915 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31634.log 2024-09-13T11:59:32.915 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38118.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36883.log.gz 2024-09-13T11:59:32.915 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39598.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-13T11:59:32.915 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.31634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38221.log 2024-09-13T11:59:32.915 INFO:teuthology.orchestra.run.smithi122.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.38118.log.gz 90.3% -- replaced with /var/log/ceph/ceph-client.admin.39598.log.gz 2024-09-13T11:59:32.915 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-mon.b.log: 2024-09-13T11:59:32.915 INFO:teuthology.orchestra.run.smithi122.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.31634.log.gz 2024-09-13T11:59:32.915 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37019.log 2024-09-13T11:59:32.915 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35230.log 2024-09-13T11:59:32.916 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38221.log: /var/log/ceph/ceph-client.admin.37019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37742.log 2024-09-13T11:59:32.916 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31594.log 2024-09-13T11:59:32.916 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38221.log.gz/var/log/ceph/ceph-client.admin.37742.log: 2024-09-13T11:59:32.916 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37019.log.gz 2024-09-13T11:59:32.916 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36608.log 2024-09-13T11:59:32.916 INFO:teuthology.orchestra.run.smithi122.stderr: 89.6%/var/log/ceph/ceph-client.admin.31594.log: -- replaced with /var/log/ceph/ceph-client.admin.35230.log.gz 2024-09-13T11:59:32.916 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38360.log 2024-09-13T11:59:32.916 INFO:teuthology.orchestra.run.smithi122.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.37742.log.gz 2024-09-13T11:59:32.916 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36298.log 2024-09-13T11:59:32.916 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31594.log.gz/var/log/ceph/ceph-client.admin.36608.log: 2024-09-13T11:59:32.916 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34803.log 2024-09-13T11:59:32.916 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36608.log.gzgzip 2024-09-13T11:59:32.916 INFO:teuthology.orchestra.run.smithi122.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35194.log 2024-09-13T11:59:32.917 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36298.log: /var/log/ceph/ceph-client.admin.34803.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.38360.log.gz 2024-09-13T11:59:32.917 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34553.log 2024-09-13T11:59:32.917 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34803.log.gz 2024-09-13T11:59:32.917 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36298.log.gz 2024-09-13T11:59:32.917 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38427.log 2024-09-13T11:59:32.917 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.34553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35552.log 2024-09-13T11:59:32.917 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39184.log 2024-09-13T11:59:32.917 INFO:teuthology.orchestra.run.smithi122.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35194.log.gz 2024-09-13T11:59:32.917 INFO:teuthology.orchestra.run.smithi122.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.34553.log.gz 2024-09-13T11:59:32.917 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35552.log: gzip -5 --verbose -- 89.5% /var/log/ceph/ceph-client.admin.36607.log 2024-09-13T11:59:32.917 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38427.log.gz 2024-09-13T11:59:32.918 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39184.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35552.log.gz 2024-09-13T11:59:32.918 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39562.log 2024-09-13T11:59:32.918 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39495.log 2024-09-13T11:59:32.918 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37947.log 2024-09-13T11:59:32.918 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39184.log.gz 2024-09-13T11:59:32.918 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39495.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36607.log.gz 2024-09-13T11:59:32.918 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34588.log 2024-09-13T11:59:32.918 INFO:teuthology.orchestra.run.smithi122.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39562.log.gz 2024-09-13T11:59:32.918 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36813.log 2024-09-13T11:59:32.918 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.34588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38222.log 2024-09-13T11:59:32.918 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37947.log.gz 2024-09-13T11:59:32.919 INFO:teuthology.orchestra.run.smithi122.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.39495.log.gz 2024-09-13T11:59:32.919 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35087.log 2024-09-13T11:59:32.919 INFO:teuthology.orchestra.run.smithi122.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34588.log.gz 2024-09-13T11:59:32.919 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35446.log 2024-09-13T11:59:32.919 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35087.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38222.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39254.log 2024-09-13T11:59:32.919 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:59:32.919 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36813.log.gz 2024-09-13T11:59:32.919 INFO:teuthology.orchestra.run.smithi122.stderr:gzip/var/log/ceph/ceph-client.admin.35446.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35767.log 2024-09-13T11:59:32.919 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37948.log 2024-09-13T11:59:32.919 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35087.log.gz 2024-09-13T11:59:32.920 INFO:teuthology.orchestra.run.smithi122.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.35446.log.gz 2024-09-13T11:59:32.920 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35767.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39254.log.gz 2024-09-13T11:59:32.920 INFO:teuthology.orchestra.run.smithi122.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.35767.log.gz 2024-09-13T11:59:32.920 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35980.log 2024-09-13T11:59:32.920 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37637.log 2024-09-13T11:59:32.920 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36470.log 2024-09-13T11:59:32.920 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37637.log: 90.8%gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-13T11:59:32.920 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35980.log.gz 2024-09-13T11:59:32.920 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37948.log.gz 2024-09-13T11:59:32.920 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36470.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37637.log.gz 2024-09-13T11:59:32.920 INFO:teuthology.orchestra.run.smithi122.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36470.log.gz 2024-09-13T11:59:32.921 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38564.log 2024-09-13T11:59:32.921 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37124.log 2024-09-13T11:59:32.921 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39563.log 2024-09-13T11:59:32.921 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37295.log 2024-09-13T11:59:32.921 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39563.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38564.log.gz 2024-09-13T11:59:32.921 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39563.log.gz 2024-09-13T11:59:32.921 INFO:teuthology.orchestra.run.smithi122.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.37124.log.gz 2024-09-13T11:59:32.921 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37845.log 2024-09-13T11:59:32.921 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34873.log 2024-09-13T11:59:32.922 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37295.log: gzip/var/log/ceph/ceph-client.admin.37845.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35981.log 2024-09-13T11:59:32.922 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.34873.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37295.log.gz 2024-09-13T11:59:32.922 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35017.log 2024-09-13T11:59:32.922 INFO:teuthology.orchestra.run.smithi122.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.37845.log.gz 2024-09-13T11:59:32.922 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34873.log.gz 2024-09-13T11:59:32.922 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35981.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.35981.log.gz 2024-09-13T11:59:32.922 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35017.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35017.log.gz 2024-09-13T11:59:32.923 INFO:teuthology.orchestra.run.smithi122.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-13T11:59:32.949 INFO:teuthology.orchestra.run.smithi100.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-13T11:59:33.042 INFO:teuthology.orchestra.run.smithi100.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-13T11:59:33.050 INFO:teuthology.orchestra.run.smithi122.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-13T11:59:34.822 INFO:teuthology.orchestra.run.smithi100.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-13T11:59:35.229 INFO:teuthology.orchestra.run.smithi122.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-13T11:59:35.270 INFO:teuthology.orchestra.run.smithi100.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-13T11:59:35.297 INFO:teuthology.orchestra.run.smithi122.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-13T11:59:35.328 INFO:teuthology.orchestra.run.smithi100.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-13T11:59:35.330 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T11:59:35.330 INFO:teuthology.orchestra.run.smithi100.stderr:real 0m2.662s 2024-09-13T11:59:35.330 INFO:teuthology.orchestra.run.smithi100.stderr:user 0m7.740s 2024-09-13T11:59:35.330 INFO:teuthology.orchestra.run.smithi100.stderr:sys 0m0.352s 2024-09-13T11:59:35.361 INFO:teuthology.orchestra.run.smithi122.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-13T11:59:35.363 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T11:59:35.363 INFO:teuthology.orchestra.run.smithi122.stderr:real 0m2.653s 2024-09-13T11:59:35.363 INFO:teuthology.orchestra.run.smithi122.stderr:user 0m7.852s 2024-09-13T11:59:35.363 INFO:teuthology.orchestra.run.smithi122.stderr:sys 0m0.429s 2024-09-13T11:59:35.363 INFO:tasks.ceph:Archiving logs... 2024-09-13T11:59:35.364 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903450/remote/smithi100/log 2024-09-13T11:59:35.365 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T11:59:36.752 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903450/remote/smithi122/log 2024-09-13T11:59:36.753 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T11:59:38.277 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-13T11:59:38.286 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-13T11:59:38.287 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T11:59:38.289 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T11:59:38.387 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-13T11:59:38.396 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-13T11:59:38.397 DEBUG:teuthology.orchestra.run.smithi100:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:59:38.399 DEBUG:teuthology.orchestra.run.smithi122:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:59:38.414 INFO:teuthology.orchestra.run.smithi100.stderr:bash: line 1: ntpq: command not found 2024-09-13T11:59:38.417 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-09-13T11:59:38.419 INFO:teuthology.orchestra.run.smithi100.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T11:59:38.419 INFO:teuthology.orchestra.run.smithi100.stdout:=============================================================================== 2024-09-13T11:59:38.419 INFO:teuthology.orchestra.run.smithi100.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 16 -36us[+1700ns] +/- 61ms 2024-09-13T11:59:38.419 INFO:teuthology.orchestra.run.smithi100.stdout:^* hv02.front.sepia.ceph.com 3 6 377 15 +713us[ +750us] +/- 59ms 2024-09-13T11:59:38.419 INFO:teuthology.orchestra.run.smithi100.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 14 -610us[ -610us] +/- 61ms 2024-09-13T11:59:38.419 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:59:38.421 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T11:59:38.421 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-09-13T11:59:38.421 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 15 -36us[ -27us] +/- 61ms 2024-09-13T11:59:38.422 INFO:teuthology.orchestra.run.smithi122.stdout:^* hv02.front.sepia.ceph.com 3 6 377 13 +714us[ +724us] +/- 59ms 2024-09-13T11:59:38.422 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 13 -620us[ -620us] +/- 61ms 2024-09-13T11:59:38.422 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:59:38.422 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-13T11:59:38.432 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-13T11:59:38.432 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-13T11:59:38.475 DEBUG:teuthology.orchestra.run.smithi100:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T11:59:38.545 DEBUG:teuthology.orchestra.run.smithi122:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T11:59:38.612 DEBUG:teuthology.orchestra.run.smithi100:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T11:59:38.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:59:38.642 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T11:59:38.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:59:38.671 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-13T11:59:38.680 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-13T11:59:38.708 INFO:teuthology.task.internal:Duration was 829.679626 seconds 2024-09-13T11:59:38.708 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-13T11:59:38.758 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-13T11:59:38.759 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T11:59:38.761 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T11:59:38.800 INFO:teuthology.orchestra.run.smithi100.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T11:59:38.802 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T11:59:39.105 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-13T11:59:39.105 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi100.front.sepia.ceph.com 2024-09-13T11:59:39.105 DEBUG:teuthology.orchestra.run.smithi100:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T11:59:39.127 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T11:59:39.127 DEBUG:teuthology.orchestra.run.smithi122:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T11:59:39.173 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-13T11:59:39.173 DEBUG:teuthology.orchestra.run.smithi100:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T11:59:39.176 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T11:59:39.242 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-13T11:59:39.242 DEBUG:teuthology.orchestra.run.smithi100:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T11:59:39.245 DEBUG:teuthology.orchestra.run.smithi122:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T11:59:39.386 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-13T11:59:39.396 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-13T11:59:39.396 DEBUG:teuthology.orchestra.run.smithi100:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T11:59:39.421 DEBUG:teuthology.orchestra.run.smithi122:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T11:59:39.447 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-13T11:59:39.460 DEBUG:teuthology.orchestra.run.smithi100:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:59:39.463 DEBUG:teuthology.orchestra.run.smithi122:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:59:39.485 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern = core 2024-09-13T11:59:39.510 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = core 2024-09-13T11:59:39.545 DEBUG:teuthology.orchestra.run.smithi100:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:59:39.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:59:39.587 DEBUG:teuthology.orchestra.run.smithi122:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:59:39.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:59:39.602 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-13T11:59:39.615 INFO:teuthology.task.internal:Transferring archived files... 2024-09-13T11:59:39.615 DEBUG:teuthology.misc:Transferring archived files from smithi100:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903450/remote/smithi100 2024-09-13T11:59:39.616 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T11:59:39.660 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903450/remote/smithi122 2024-09-13T11:59:39.660 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T11:59:39.692 INFO:teuthology.task.internal:Removing archive directory... 2024-09-13T11:59:39.692 DEBUG:teuthology.orchestra.run.smithi100:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T11:59:39.697 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T11:59:39.742 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-13T11:59:39.753 INFO:teuthology.task.internal:Not uploading archives. 2024-09-13T11:59:39.753 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-13T11:59:39.793 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-13T11:59:39.793 DEBUG:teuthology.orchestra.run.smithi100:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T11:59:39.796 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T11:59:39.808 INFO:teuthology.orchestra.run.smithi100.stdout: 265340 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 11:59 /home/ubuntu/cephtest 2024-09-13T11:59:39.811 INFO:teuthology.orchestra.run.smithi122.stdout: 266063 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 11:59 /home/ubuntu/cephtest 2024-09-13T11:59:39.813 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-13T11:59:39.843 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-13T11:59:39.945 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest} thrashers/force-sync-many workloads/pool-create-delete} duration: 829.6796264648438 flavor: default owner: scheduled_akupczyk@teuthology success: true 2024-09-13T11:59:39.945 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T11:59:40.034 INFO:teuthology.run:pass